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The agreement continued the collaborative work between the Appalachian Fruit Research Station 
and Cornell University with techniques to segment fresh and stored apple bruising on the same 
apple (Throop et. al. 1995). Studies for either positioning the stem/calyx ends in a known position 
by orientation (Throop et. al. 1995, Throop et. al. 1997, Throop et. al. 1999) or identifying the stem 
and calyx (Campins et. al. 1997) to avoid detection as a defect were pursued. Spectra to determine 
the reflectance properties of the many different defects that can occur on the different cultivars that 
grow in Eastern United States were recorded. Statistical and neural network analysis was used to 
find the best wavelengths for identifying 22 different defects on 10 cultivars (Aneshansley et. al. 
1997, Miller et. al. 1998). Methods to segment defects were developed for each of the identified 
wavelengths or combination of wavelengths (Throop and Aneshansley, 1997a and 1997b). 
Orientation, reflectance spectra, and methods of segmentation were incorporated into an inspection 
station to find defects on apples at a rate of 5 apples per second (Throop and Aneshansley, 1999a 
and 1999b). With the possibility of orientation, investigations using body transmittance to detect 
internal browning were conducted (Upchurch et. al. 1996). Further work needs to be done to bring 
this work on line. 


Papers and patents associated with this project are listed below: 


Throop, J. A., D. J. Aneshansley, and B. L. Upchurch. 1995. An image processing algorithm to find 
new and stored apple bruises. Applied Engineering in Agriculture of the American Society of 
Agricultural Engineers. 11(5):751-757. 


Throop, J. A., D. J. Aneshansley, and B. L. Upchurch. 1995. Apple orientation on automatic sorting 
equipment. ASAE Paper 956176. ASAE, St. Joseph, MI 49085-9659 


Throop, J. A., D. J. Aneshansley, B. L. Upchurch. 1997. Apple orientation on automatic sorting 
equipment. Proceedings from the Sensors for Nondestructive Testing International Conference, 
Orlando Florida. pp324-342. 


Aneshansley, D. J., J. A. Throop, and B. L. Upchurch. 1997. Reflectance spectra of surface defects 
on apples. Proceedings from the Sensors for Nondestructive Testing International Conference, 
Orlando Florida. pp143-160. 


Campins, J., J. A. Throop, and D. J. Aneshansley. 1997. Apple stem and calyx identification for 
automatic sorting. ASAE Paper 973079. ASAE, St. Joseph, MI 49085-9659 
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Throop, J. A., D. J. Aneshansley. 1997a. Inspection of processed fruit before and after peeling. 
ASAE Paper 976044. ASAE, St. Joseph, MI 49085-9659 


Throop, J. A., D. J. Aneshansley. 1997b. Apple Damage Segmentation utilizing reflectance spectra 
of the defect. ASAE Paper 973078. ASAE, St. Joseph, MI 49085-9659 


Upchurch, B. L., J. A. Throop, and D. J. Aneshansley, 1997. Detecting internal breakdown in apples 
using interactance measurements. Postharvest Biology and Technology 10(1997)15-19. 


Miller, W. M., J. A. Throop, and B. L. Upchurch. 1998. Pattern recognition models for spectral 
reflectance evaluation of apple blemishes. Postharvest Biology and Technology 14(1998)1 1-20. 


Throop, J. A., D. J. Aneshansley, and B. L. Upchurch. 1999a. Fruit orienting device. U. S. Patent 
No. 5,855,270. 19p. Commissioner of Patents and Trade Marks, Washington, D.C. 


Throop, J. A., D. J. Aneshansley, and B. Anger. 1999b. Inspection station detects defects on apples 
in real time. ASAE Paper 993205. ASAE, St. Joseph, MI 49085-9659 
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AN IMAGE PROCESSING ALGORITHM TO 
FIND NEw AND OLp BRUISES 


J. A. Throop, D. J. Aneshansley, B. L. Upchurch 


ABSTRACT. Stored 


d Delicious’ apples can have old and new bruises. Previous studies have shown the near-infrared 


‘Re 
(NIR) reflection of bruised tissue with respect to unbruised tissue to be darker in new (24 h) bruises and lighter in old 
(four to six weeks) bruises. An algorithm was developed to identify both old and new bruises in the same NIR image. 
Apple images were low pass filtered to remove small surface characteristics (lenticels) and normalized. Absolute 
difference between these images and a standard (delrin sphere) representing unbruised tissue were also low pass filtered. 
A binary image of bruised/ unbruised tissue was formed based upon a user defined threshold level. Shapes of the bruised 
areas were evaluated and, based upon a shape factor, either dismissed or counted as bruised areas. The algorithm 


detected both old and new bruises except for large 25 to 35 m 
areas within their boundaries where NIR reflectance had not 


m (0.98 to 1.38 in.) diameter old bruises. These bruises had 


changed completely from darker to lighter than unbruised 


tissue. This patterning altered the bruise shape causing the algorithm to discount them, Keywords. Apples, Apple quality, 


Bruise, Bruise detection, Image analysis, Image processing. 


lgorithms to separate bruise information from 

the rest of the NIR image of an apple have been 

attempted by only a few researchers. Digital 

Matrix camera images of apple NIR reflectance 
have been processed with an algorithm using a 
multivariable linear statistical Classifier that classified 
bruised from unbruised apple tissue as well as a human 
inspector (Graf et al., 1981). 

A digital linescan camera captured images of apple NIR 
reflectance eliminating both spatial distortion in one 
dimension and image background (Taylor, 1985). However, 
the pixel grey level nonuniformity caused data variability 
requiring a time-consuming data normalization process for 
a quadratic statistical classifier to operate. 

A nonstatistical algorithm for linescan images of apple 
NIR reflectance was developed (Throop et al., 1988) and 
tested on experimentally machine-harvested apples 
(Rehkugler et al., 1989). This algorithm creates a new 
image from the original by averaging a plurality of 
adjacent pixel values of the same row. The original image 
is subtracted from the averaged image. The resulting image 
is inverted and converted to a binary image by 
thresholding. The black pixel clusters are evaluated for 


Se 
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their shape by finding the relationship of their area with 
respect to their perimeter (shape factor). Only elliptical- to 
Circular-shaped clusters (shape factor 0.7 to 1.5) are 
counted as bruises. 

The above algorithm was only applied to apple images 
of NIR reflectance which were bruised and held at room 
temperature for 24 h. NIR reflectance has been shown to 
change with extended Storage (Rehkugler et al., 1976; 
Upchurch et al., 1992). NIR reflectance of bruised tissue is 
less than‘unbruised tissue 24 h after bruising, but slowly 
increases to be equal and exceed the reflectance of 
unbruised tissue over time (one to two months) during cold 
Storage. The problem of Sorting fruit which could have 
both old (stored) bruises and new bruises, acquired during 
removal from storage, has not been addressed. 


OBJECTIVES 

The first objective is to develop an algorithm which can 
separate old and new bruised tissue from stored apples by 
using NIR reflectance. The second Objective is to 
demonstrate an experimental method to estimate the value 
of user supplied variables, minimum shape factor, and 
threshold factor required in the algorithm for optimal 
bruised tissue classification. 


MATERIALS AND METHODS 

A 512 diode array line scan camera (EG&G Reticon 
LC1901FAN-011, Sunnyvale, Calif.) with a 25 mm 
(0.98 in.) lens using an aperture setting of f2.0 and with a 
longpass filter (Wratten 89B, Kodak, Rochester, N.Y.) in 
front of the lens captured the images. Image digitization 
was performed by a line scan processor (DT2856, Data 
Translation, Marlboro, Md.) set for externally triggered 
Single line per trigger acquisition mode, clocked at 
156.25 kHz, 525 counts per line, a line transfer value of 20, 
and a total line count of 360. The intensity value for every 
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five counts per line were averaged so that a 360V x 105H 
pixel image of the apple surface was formed. The 
microcomputer (Gateway 386SX/25 mHz, North Sioux 
City, S.D.) provided a pulse to trigger both the stepper 
motor and the line scan camera. Subroutine functions 
(DT_Linescan, Data Translation) and the “C” 
programming language (Microsoft C 6.0, Microsoft Corp., 
Redmond, Wash.) were used for algorithm development. 

Illumination consisted of eight standard incandescent 
lamps (model 40A, 40W, General Electric, Cleveland, 
Ohio) placed between an outer reflective chamber surface 
and an inner white acrylic diffuser [254 um (10 mil)]. 
Apples were rotated by a stepper motor (ABS 3008-006, 
12V, 1.0°, Hurst, Princeton, Ind.) driven spindle on the 
opposite side of the acrylic diffuser. 

A custom turned 7.62 cm (3.0 in.) diameter white delrin 
sphere (Great Lakes Plastics, Buffalo, N.Y.) is known to be 
a good standard to represent unbruised ‘Red Delicious’ 
apple tissue (Throop et al., 1993). 


ALGORITHM DESCRIPTION 
LOAD AND PROCESS IMAGE OF DELRIN SPHERE 

The delrin sphere image (fig. 1c), captured for the same 
lighting and camera setup as for capturing the apple images 
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(figs. 1a and 1b), is normalized to a grey level intensity of 
200. The average intensity of each 360 pixel column is 
found and divided into 200. The product of the resulting 
scaling factor and each pixel value in that column is 
substituted in the delrin sphere image (fig. 1d). 


LoaD AND Low Pass FILTER APPLE IMAGE 

Each apple image (figs. 1a and 1b) is loaded and a 5 x 5 
mean filter convolution is performed. The need for this 
process was to smooth the intensities of the brighter 
lenticels visible in the image background so that a more 
uniform background average can be found for background 
normalization. 


NORMALIZE BACKGROUND OF APPLE IMAGE 

The sum of the center column pixel intensities of the 
upper 180 pixel rows is found. All columns of 180 pixel 
rows whose sum is equal or greater than 80% of the central 
column sum are normalized. The remaining pixels are set 
to 0. The average pixel intensity for each 180 pixel column 
to be normalized is found and divided into 200. Each pixel 
intensity for that column is multiplied by the resulting scale 
factor (fig. 2a). A similar operation is performed on rows 
181 to 360 resulting in the apple image background 
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Figure 1-Composite NIR grey tone digital linescan image: (a) apple with 24-h-old bruise, (b) apple with 2-month-old bruise, (c) delrin sphere 


unnormalized, and (d) delrin sphere normalized to grey level 200. 
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(fig. 2b) being of approximately equal intensity as the 
normalized delrin sphere standard (fig. 1d). 


SUBTRACT APPLE IMAGE FROM 
NORMALIZED DELRIN IMAGE 

By normalizing the backgrounds of both the apple and 
delrin images, their absolute difference results in an image 
of the high frequency intensity changes, brighter and 
darker, appearing as light pixels. Each pixel of the resulting 
image is multiplied by 10 (fig. 2c). 


Low Pass FILTER THE RESULTING IMAGE 

The image is smoothed by a 5 x 5 mean filter 
convolution to even out any small pixel areas of sharp 
contrast. This causes a slight blurring which is visible in 
figure 2d. 


THRESHOLD IMAGE TO CONVERT TO A BINARY IMAGE 

The average grey level for the first 180 rows of each’ 
column is calculated and multiplied by a user supplied 
multiplier (threshold factor). Any pixel grey level greater 
than this value is set to 0, otherwise the pixel is set to 255 
(fig. 3a). This process is repeated on each column for rows 
181 to 360. The resulting binary image has black pixels at 
all of the previous brighter areas and white pixels for the 


darker background (fig. 3b). The threshold factor value can 
be estimated by the user processing a small representative 
Sample of bruised fruit. Examination of the resulting 
thresholded images to determine if the bruised areas have 
been segmented from the surrounding unbruised tissue can 


be used as a measure to approximate the value of the 
threshold factor. 


CLOSING OF THE IMAGE 

The mathematical morphological process 
(Haralick et al., 1987) of closing black pixel objects with a 
5 x 5 structuring element eliminated details in the image 
that were smaller than the structuring element size and 
produced little change in the size of larger black pixel 
clusters (fig. 3c), 


FIND PIXEL CLUSTER SHAPE FROM AREA AND PERIMETER 

The area and perimeter of each black pixel cluster in 
figure 3c is found and the cluster shape factor is found 
(Rehkugler et al., 1989). If the shape factor (1.0 for a 
circle) is greater than a user selected minimum and less 
than 1.5, the pixel cluster is counted as a bruise and the 
area is recorded; otherwise, the algorithm discounts the 
dark cluster as a bruise and continues. The final processed 





(a) (b) 


(c) (d) 


Figure 2—Processed NIR images of an apple with a 24-h-old bruised apple: (a) background normalized to grey level equal to 200 for half of the 
apple surface, (b) all of background normalized to grey level 200, (c) absolute difference of normalized sphere and apple image, and (d) 5 x 5 


low pass filtered image of c. 


VoL. 11(5):751-757 


image shows that the algorithm has successfully segmented 
the bruised area from the unbruised areas (fig. 3d). 


LOAD A NEW APPLE IMAGE AND REPEAT IMAGE PROCESS 

Figure 1a is a NIR image of a 24-h-old bruise on a ‘Red 
Delicious’ apple. Figure 1b is the same apple bruise after 
two months in storage. Figures 4 and 5 show a sequence of 
images as processing occurs for the two-month-old bruise. 
The results are very similar to the same apple 24 h after 
bruising (figs. 2 and 3) except that the final processed 
image shows three small false clusters that were counted as 
bruises (fig. 5d). 


SELECTION OF THRESHOLD AND SHAPE 
FACTORS IN THE ALGORITHM 


Eighty-seven ‘Red Delicious’ apples, harvested during 
the 1992 harvest season, were used to evaluate the 
algorithm. The apples were removed from cold storage in 
December 1992, and allowed to warm to room temperature 
(6 h). The apples were divided into three groups of 29 
apples. Each apple of the first group was dropped 5 cm 
(1.97 in.) [bruise diameters = 11 to 15 mm (4.33 to 
5.91 in.)], the second group was dropped 15 cm (5.91 in.) 





(a) (b) 


[bruise diameters = 19 to 25 mm (7.48 to 9.84 in.)], and the 
third group was dropped 25 cm (9.84 in.) [bruise diameters 
= 25 to 35 mm (9.84 to 13.77 in.)] onto a flat steel plate. 
Images were captured 24 h and 2 months after bruising 
with the apples held in cold storage between image 
captures. 

The algorithm was used to process the three groups of 
‘Red Delicious’ apples using three different threshold 
factors (1.2, 1.4, 1.6) and three different minimum shape 
factors (0.8, 0.9, 1.0). 


RESULTS AND DISCUSSION 

A number of pixel clusters which were not bruise pixels 
were retained after the algorithm processed the apple 
images 24 h after bruising. Table 1 shows the average 
number of false bruise pixel clusters per apple and their 
average area in pixels which were included as bruised 
tissue for three drop heights, three minimum shape factors, 
and three threshold factors. For any minimum shape factor 
selected, as the threshold factor increased, the average 
number of false bruise pixel clusters decreased. Also for 
any threshold factor selected, as the minimum. shape factor 
increased, the number of false bruise pixel clusters 
decreased. 


(c) (d) 


Figure 3-Further processing of an apple with a 24-h-old bruise: (a) binary threshold of top half of image based on the product of column 
average and threshold factor, (b) binary threshold of bottom half of image based on the product of column average and threshold factor, 
(c) closed binary image, and (d) pixel cluster counted as a bruise based on shape. 
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AN IMAGE PROCESSING ALGORITHM TO 
FIND NEW AND OLD BRUISES 


J.A. Throop, D. J. Aneshansley, B. L. Upchurch 


ABSTRACT. Stored ‘Red Delicious’ apples can have old and new bruises. Previous studies have shown the near-infrared 
(NIR) reflection of bruised tissue with respect to unbruised tissue to be darker in new (24 h) bruises and lighter in old 
(four to six weeks) bruises. An algorithm was developed to identify both old and new bruises in the same NIR image. 
Apple images were low pass filtered to remove small surface characteristics (lenticels) and normalized. Absolute 
difference between these images and a standard (delrin sphere) representing unbruised tissue were also low pass filtered. 
A binary image of bruised/unbruised tissue was formed based upon a user defined threshold level. Shapes of the bruised 
areas were evaluated and, based upon a shape factor, either dismissed or counted as bruised areas. The algorithm 
detected both old and new bruises except for large 25 to 35 mm (0.98 to 1.38 in.) diameter old bruises. These bruises had 
areas within their boundaries where NIR reflectance had not changed completely from darker to lighter than unbruised 
tissue. This patterning altered the bruise shape causing the algorithm to discount them. Keywords. Apples, Apple quality, 


Bruise, Bruise detection, Image analysis, Image processing. 


lgorithms to separate bruise information from 

the rest of the NIR image of an apple have been 

attempted by only a few researchers. Digital 

matrix camera images of apple NIR reflectance 
have been processed with an algorithm using a 
multivariable linear statistical classifier that classified 
bruised from unbruised apple tissue as well as a human 
inspector (Graf et al., 1981). 

A digital linescan camera captured images of apple NIR 
reflectance eliminating both spatial distortion in one 
dimension and image background (Taylor, 1985). However, 
the pixel grey level nonuniformity caused data variability 
requiring a time-consuming data normalization process for 
a quadratic statistical classifier to operate. 

A nonstatistical algorithm for linescan images of apple 
NIR reflectance was developed (Throop et al., 1988) and 
tested on experimentally machine-harvested apples 
(Rehkugler et al., 1989). This algorithm creates a new 
image from the original by averaging a plurality of 
adjacent pixel values of the same row. The original image 
is subtracted from the averaged image. The resulting image 
is inverted and converted to a binary image by 
thresholding. The black pixel clusters are evaluated for 
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their shape by finding the relationship of their area with 
respect to their perimeter (shape factor). Only elliptical- to 
circular-shaped clusters (shape factor 0.7 to 1.5) are 
counted as bruises. 

The above algorithm was only applied to apple images 
of NIR reflectance which were bruised and held at room 
temperature for 24 h. NIR reflectance has been shown to 
change with extended storage (Rehkugler et al., 1976; 
Upchurch et al., 1992). NIR reflectance of bruised tissue is 
less than‘unbruised tissue 24 h after bruising, but slowly 
increases to be equal and exceed the reflectance of 
unbruised tissue over time (one to two months) during cold 
storage. The problem of sorting fruit which could have 
both old (stored) bruises and new bruises, acquired during 
removal from storage, has not been addressed. 


OBJECTIVES 

The first objective is to develop an algorithm which can 
separate old and new bruised tissue from stored apples by 
using NIR reflectance. The second objective is to 
demonstrate an experimental method to estimate the value 
of user supplied variables, minimum shape factor, and 
threshold factor required in the algorithm for optimal 
bruised tissue classification. 


MATERIALS AND METHODS 

A 512 diode array line scan camera (EG&G Reticon 
LC1901FAN-011, Sunnyvale, Calif.) with a 25 mm 
(0.98 in.) lens using an aperture setting of f2.0 and with a 
longpass filter (Wratten 89B, Kodak, Rochester, N.Y.) in 
front of the lens captured the images. Image digitization 
was performed by a line scan processor (DT2856, Data 
Translation, Marlboro, Md.) set for externally triggered 
single line per trigger acquisition mode, clocked at 
156.25 kHz, 525 counts per line, a line transfer value of 20, 
and a total line count of 360. The intensity value for every 
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five counts per line were averaged so that a 360V x 105H 
pixel image of the apple surface was formed. The 
microcomputer (Gateway 386SX/25 mHz, North Sioux 
City, S.D.) provided a pulse to trigger both the stepper 
motor and the line scan camera. Subroutine functions 
(DT 2Linescan, Data translation) fandsunes ec. 
programming language (Microsoft C 6.0, Microsoft Corp., 
Redmond, Wash.) were used for algorithm development. 

Illumination consisted of eight standard incandescent 
lamps (model 40A, 40W, General Electric, Cleveland, 
Ohio) placed between an outer reflective chamber surface 
and an inner white acrylic diffuser [254 um (10 mil)]. 
Apples were rotated by a stepper motor (ABS 3008-006, 
12V, 1.0°, Hurst, Princeton, Ind.) driven spindle on the 
opposite side of the acrylic diffuser. 

A custom turned 7.62 cm (3.0 in.) diameter white delrin 
sphere (Great Lakes Plastics, Buffalo, N.Y.) is known to be 
a good standard to represent unbruised ‘Red Delicious’ 
apple tissue (Throop et al., 1993). 


ALGORITHM DESCRIPTION 
LOAD AND PROCESS IMAGE OF DELRIN SPHERE 

The delrin sphere image (fig. 1c), captured for the same 
lighting and camera setup as for capturing the apple images 


3S 
2. 


RA SAA 


wa 
S 


ASN 


3 


REAR ae Un 
BN RARSAY. RTAeaNeS 
ARRAN AAA 


SS Yee Sete: 


ARAN 


DESL ALL AC CAPO AAD ON OCR SAB ODIO A 


OR 


. 
: 
i 
: 
z 
S 
z 
= 
% 
3: 


as 
ERS 


Aan 


eAAN A 


ARAN AAA AOAAAA NOOR 


NSAAN SAAS 


WANNA OOOO ~ 
SAAN 


SRUAN NSA Se ANS SS SRN VS NE SNS Ve Na eve NSN 


SSSA 


PONS 


SSS NSA SS SA ON SN NNT TN 


NSS 


SAAN AN 
SESS 


AS 
ANS: 


SS 
NSASS: 


nS 





(figs. 1a and 1b), is normalized to a grey level intensity of 
200. The average intensity of each 360 pixel column is 
found and divided into 200. The product of the resulting 
scaling factor and each pixel value in that column is 
substituted in the delrin sphere image (fig. 1d). 


LOAD AND Low PASS FILTER APPLE IMAGE 

Each apple image (figs. 1a and 1b) is loaded and a 5 x 5 
mean filter convolution is performed. The need for this 
process was to smooth the intensities of the brighter 
lenticels visible in the image background so that a more 
uniform background average can be found for background 
normalization. 


NORMALIZE BACKGROUND OF APPLE IMAGE 

The sum of the center column pixel intensities of the 
upper 180 pixel rows is found. All columns of 180 pixel 
rows whose sum is equal or greater than 80% of the central 
column sum are normalized. The remaining pixels are set 
to 0. The average pixel intensity for each 180 pixel column 
to be normalized is found and divided into 200. Each pixel 
intensity for that column is multiplied by the resulting scale 
factor (fig. 2a). A similar operation is performed on rows 
181 to 360 resulting in the apple image background 
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Figure 1-Composite NIR grey tone digital linescan image: (a) apple with 24-h-old bruise, (b) apple with 2-month-old bruise, (c) delrin sphere 


unnormalized, and (d) delrin sphere normalized to grey level 200. 
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(fig. 2b) being of approximately equal intensity as the 
normalized delrin sphere standard (fig. 1d). 


SUBTRACT APPLE IMAGE FROM 
NORMALIZED DELRIN IMAGE 

By normalizing the backgrounds of both the apple and 
delrin images, their absolute difference results in an image 
of the high frequency intensity changes, brighter and 
darker, appearing as light pixels. Each pixel of the resulting 
image is multiplied by 10 (fig. 2c). 


Low PAss FILTER THE RESULTING IMAGE 

The image is smoothed by a 5 xX 5 mean filter 
convolution to even out any small pixel areas of sharp 
contrast. This causes a slight blurring which is visible in 
figure 2d. 


THRESHOLD IMAGE TO CONVERT TO A BINARY IMAGE 

The average grey level for the first 180 rows of each’ 
column is calculated and multiplied by a user supplied 
multiplier (threshold factor). Any pixel grey level greater 
than this value is set to 0, otherwise the pixel is set to 255 
(fig. 3a). This process is repeated on each column for rows 
181 to 360. The resulting binary image has black pixels at 
all of the previous brighter areas and white pixels for the 


darker background (fig. 3b). The threshold factor value can 
be estimated by the user processing a small representative 
sample of bruised fruit. Examination of the resulting 
thresholded images to determine if the bruised areas have 
been segmented from the surrounding unbruised tissue can 
be used as a measure to approximate the value of the 
threshold factor. 


CLOSING OF THE IMAGE 

The mathematical morphological process 
(Haralick et al., 1987) of closing black pixel objects with a 
5 x 5 structuring element eliminated details in the image 
that were smaller than the structuring element size and 
produced little change in the size of larger black pixel 
clusters (fig. 3c). 


FIND PIXEL CLUSTER SHAPE FROM AREA AND PERIMETER 

The area and perimeter of each black pixel cluster in 
figure 3c is found and the cluster shape factor is found 
(Rehkugler et al., 1989). If the shape factor (1.0 for a 
circle) is greater than a user selected minimum and less 
than 1.5, the pixel cluster is counted as a bruise and the 
area is recorded; otherwise, the algorithm discounts the 
dark cluster as a bruise and continues. The final processed 





(a) (b) 


(c) (d) 


Figure 2—Processed NIR images of an apple with a 24-h-old bruised apple: (a) background normalized to grey level equal to 200 for half of the 
apple surface, (b) all of background normalized to grey level 200, (c) absolute difference of normalized sphere and apple image, and (d) 5 x 5 


low pass filtered image of c. 
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image shows that the algorithm has successfully segmented 
the bruised area from the unbruised areas (fig. 3d). 


LOAD A NEW APPLE IMAGE AND REPEAT IMAGE PROCESS 

Figure la is a NIR image of a 24-h-old bruise on a ‘Red 
Delicious’ apple. Figure 1b is the same apple bruise after 
two months in storage. Figures 4 and 5 show a sequence of 
images as processing occurs for the two-month-old bruise. 
The results are very similar to the same apple 24 h after 
bruising (figs. 2 and 3) except that the final processed 
image shows three small false clusters that were counted as 
bruises (fig. 5d). 


SELECTION OF THRESHOLD AND SHAPE 
FACTORS IN THE ALGORITHM 


Eighty-seven “Red Delicious’ apples, harvested during 
the 1992 harvest season, were used to evaluate the 
algorithm. The apples were removed from cold storage in 
December 1992, and allowed to warm to room temperature 
(6 h). The apples were divided into three groups of 29 
apples. Each apple of the first group was dropped 5 cm 
(1.97 in.) [bruise diameters = 11 to 15 mm (4.33 to 
5.91 in.)], the second group was dropped 15 cm (5.91 in.) 





ine reas — 






(a) (b) 


[bruise diameters = 19 to 25 mm (7.48 to 9.84 in.)], and the 
third group was dropped 25 cm (9.84 in.) [bruise diameters 
= 25 to 35 mm (9.84 to 13.77 in.)] onto a flat steel plate. 
Images were captured 24 h and 2 months after bruising 
with the apples held in cold storage between image 
captures. 

The algorithm was used to process the three groups of 
‘Red Delicious’ apples using three different threshold 
factors (1.2, 1.4, 1.6) and three different minimum shape 
factors (0.8, 0.9, 1.0). 


RESULTS AND DISCUSSION 

A number of pixel clusters which were not bruise pixels 
were retained after the algorithm processed the apple 
images 24 h after bruising. Table 1 shows the average 
number of false bruise pixel clusters per apple and their 
average area in pixels which were included as bruised 
tissue for three drop heights, three minimum shape factors, 
and three threshold factors. For any minimum shape factor 
selected, as the threshold factor increased, the average 
number of false bruise pixel clusters decreased. Also for 
any threshold factor selected, as the minimum. shape factor 
increased, the number of false bruise pixel clusters 
decreased. 





(c) (d) 


Figure 3-Further processing of an apple with a 24-h-old bruise: (a) binary threshold of top half of image based on the product of column 
average and threshold factor, (b) binary threshold of bottom half of image based on the product of column average and threshold factor, 
(c) closed binary image, and (d) pixel cluster counted as a bruise based on shape. 
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A false pixel area of 198 pixels translates to an 
equivalent bruise diameter of 15.9 mm (0.625 in.). This is 
equal to the maximum bruising allowed for extra fancy 
fruit (USDA Agricultural Marketing Services, 1972). All 
false bruise clusters for a minimum shape factor of 0.8 and 
false bruise clusters for a minimum shape factor of 0.9 with 
a threshold factor of 1.2 exceed a false pixel area of 200 
pixels. The value of 0.8 can be eliminated as an acceptable 
shape factor. 

Table 2 shows the percentage of bruises that were 
correctly identified by the algorithm using the two 
remaining shape factors, three drop heights, and three 
threshold factors. Using a minimum shape factor of 0.9 and 
a threshold factor of 1.4, the greatest percentage of bruises 
was found regardless of drop height (bruise size). For all 
threshold and minimum shape factors, as the drop height 
(bruise size) increased, the percent of bruises located 
increased. 

Table 3 shows the percentage of 24-h-old and 2-month- 
old bruises correctly located and the average number of 
false pixel clusters per apple using a shape factor equal to 
0.9 and threshold factor equal to 1.4. The results for both 
bruise ages are very similar except for the largest bruise 
size [25 cm (9.84 in.) drop height]. The two-month-old 
large bruises had small pixel areas within the bruise 


(a) (b) 


boundaries where the reflectance had changed to equal 
unbruised tissue, but was not greater than unbruised tissue. 
This altered the bruise shape causing the algorithm to 
discount the large bruise. The percentage of two-month-old 
large bruises [25 cm (9.84 in.) drop height] correctly 
located was 48% compared to 93% for 24-h-old bruises. 


CONCLUSIONS 

An algorithm was developed that could detect both 
24-h-old (NIR reflectance less than unbruised tissue) and 
2-month-old bruises (NIR reflectance greater than 
unbruised tissue). 

A minimum shape factor equal to 0.9 and threshold 
factor of 1.4 best separated 24-h-old and 2-month-old 
bruises with bruise diameters of 11 to 35 mm (0.43 to 
1.38 in.). The two-month-old large bruises had small pixel 
areas within the bruise boundaries where the reflectance 
had not changed completely from darker to lighter than 
unbruised tissue. This patterning altered the bruise shape 
causing the algorithm to discount the large bruise. An 
average of one false bruise cluster can be expected on each 
apple with a 200 pixel area equivalent to 15.9 mm 
(0.625 in.) diameter bruise for the 0.9 minimum shape 
factor and the 1.4 threshold factor selected. 





(c) (d) 


Figure 4—Processed NIR images of an apple with a two-month-old bruise: (a) background normalized to grey level equal to 200 for half of the 
apple surface, (b) all of background normalized to grey level 200, (c) absolute difference of normalized sphere and apple image, and (d) 5 x 5 


low pass filtered image of c. 
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Figure 5—Further processing of an apple with a two-month-old bruise: (a) binary threshold of top half of image based on the product of column 
average and threshold factor, (b) binary threshold of bottom half of image based on the product of column average and threshold factor, 
(c) closed binary image, and (d) pixel cluster counted as a bruise based on shape. 


Table 2. Average percentage of 24-h-old bruises that were correctly 
identified by the algorithm for two shape factors, three 


drop heights, and three threshold factors 
Table 1. Average number and area in pixels of error clusters for Pee 





three threshold factors, three shape factors, and three Drop Height Threshold Factors 
drop heights for apples with 24-h-old bruises oy 1.2% 1.4% 1.6% 
Threshold Factors Shape Factor = 0.9 
1.2 1.4 1.6 5 52 52 38 
Drop No. No. No. * - 


Height Error Error Error 


(cm) Clusters Area Clusters Area Clusters Area Shape Factor = 1.0 
5 








Shape Factor = 0.8 28 28 28 
5 2.9 220 2.6 166 1.8 173 15 ay 66 De 
15 2.7 245 ep 230 15 189 25 79 86 86 
25 ow 288 23 201 tea 179 
Shape Factor = 0.9 
5 i 220 11 ilsiy/ 0.6 171 Table 3. The percent of correctly identified bruises and the number of 
15 0.9 326 0.7 188 0.6 171 error clusters for 24-h-old and 2-month-old bruises for three drop 
25 1.7 347 1.1 184 1.0 172 heights using a threshold factor of 1.4 and a shape factor of 0.9 
eh a al a A SE a 
Shape Factor = 1.0 Dro anne ety eu eeaee 2 Month 
5 0.4 490 02 114 03 137 (cm) (9%) (No.) (%) (No.) 
15 0.3 571 0.2 477 0.1 156 
25 03 971 03 173 0.1 162 5 52 1 52 1. 
eee ee em 15 69 0.7 66 0. 
NOTE: 1 pixel = 1mm’ (0.00155 in.’). 25 93 1d 48 13 
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Summary: Four methods to orient apples so that their axis of 


rotation passes through the stem/calyx ends are 
presented. The time to orient unoriented apples, the 
time apples placed in orientation remain, and their 
height and width are recorded for two commercial 
conveyors and two experimental conveyors for 5 apple 
cultivars. One experimental method oriented 88% to 100% 
of 4 cultivars. Red Delicious oriented poorly on all 
devices. The difference between apple height and width, 
fruit shape, size, stem stiffness and length, and 
surface curvature appear to play a roll in apple 
orientation. 
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ABSTRACT 


Four methods to orient apples to place their axis of 
rotation through the stem and calyx were tested. Two methods used 
commercial bicone conveyors and two were experimental, consisting 
of free- to-rotate cone rollers and a powered cylindrical roller. 
The times to orient an unoriented apple and the time an apple 
placed in orientation remained in orientation were recorded. 

Experimental method 1 oriented Cortlands (100%), Empires 
(88%), Idareds (96%) and Jonagolds (98%). If orientation 
occurred, it was sucessful within 10 - 13 seconds. Furthermore, 
once oriented, they stayed oriented at least 98% percent of the 
time for 20 seconds (maximum period of observation). However, 
method 1 was unsuccessful in orienting Red Delicious (12%) and 
only 60% stayed oriented for the 20 seconds. 

Experimental method 2 required 4 - 5 seconds longer to 
orient Cortlands, Idareds, Jonagold, and Empires. As the oriented 
fruit rotated, they tended to wobble slightly about its axis of 
rotation caused by small geometric irregularities of the apple’s 
shape. With time this motion caused the oriented apple’s axis of 
rotation of some fruit to become unstable resulting in fruit 
disorientation. 

Commercial methods 3 and 4 did not orient Empires, 
Cortlands, Idareds, or Jonagolds. In fact, the axis passing 
through the stem and calyx ends consistently changed to 
perpendicular to the axis of rotation of the bicone rollers for 
the 4 cultivars. However, some Red Delicious were oriented by 
methods 3 and 4, 68% and 60% respectively, with orientation 
occurring within 16 seconds. For oriented apples, less than 30% 
of the Empires and less than 20% of Cortlands, Idareds, or 
Jonagolds remained oriented while 70% of Red Delicious stayed 
oriented for method 3 compared to 65% for method 4. 

The difference between apple height and apple width 
played a part in the orientation processes tested, but other 
factors such as fruit shape, size, stem length, stem stiffness 
and surface curvature appear to play a roll also. 


INTRODUCTION 


Considerable effort has been put into applying sensor 
techniques to nondestructively sense the internal and surface 
quality of apples. Two trains of thought regarding the handling 
of the fruit during this process have been taken: 1)Convey the 
fruit past the sensors on some form of rotating cones or rollers 
while using the sensor to locate the stem/calyx regions and any 
defect, or 2)Orient the fruit in such a manner that the locations 
of the stem/calyx regions are known and use the sensor to locate 
defects. 

Yang, (1992) demonstrated a technique using structured light 
to differentiate the concave surfaces around the stem/calyx ends 
from blemishes on the rest of the convex apple surface. Yang, 
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(1993) carried the identification process a step further by 
applying neural networks. The fruit packing industry requests 
that apples must be sorted at rates of 6 - 10 apples per second. 
There is little processing time to waste on locating the 
stem/calyx ends. If the stem/calyx ends pass directly through the 
camera view, extra lighting and mirrors or cameras must be added 
to view the sides of the apple as it rotates on bicones. 

Davenel et. al. (1988) built an automatic detection system 
for surface defects on apples using a commercial sorting ; 
conveying system. Their algorithm depended on the fruit turning 
about a horizontal axis running through the stem and calyx 
cavities on bicone rollers. During their study, video recordings 
of commercial sorters equipped with "bicone" conveyors showed 
that fruit did not rotate as their algorithm required. 

Abbott, (1994), Armstrong and Brown, (1991), Pitts et. al., 
(1991) , Peleg, .(1993)7anaschen etvealsae (19930 have all 
attempted to measure apple firmness by a number of different 
techniques all requiring knowledge of where the stem/calyx 
regions are located. Orientation is a requirement if this type of 
measurement is considered. 

Rehkugler and Throop (1976) demonstrated two apple 
orientation devices to place the stem/calyx regions in a known 
location. One device consisted of a combination of free-to-rotate 
cone rollers and a powered rotating padded cylindrical roller. 
Data on the ability of this system to orient apples will be 
reported here as the second orientation method tested in this 
paper. Another device consisted of a rotating inverted conical 
shaped transport cup with an opening at the bottom (Keesling, 
1965). Through the opening a small rotating wheel contacted the 
fruit until the stem/calyx was positioned over the wheel causing 
loss of apple contact and ending apple rotation. Rehkugler and 
Throop (1986) presented data showing the time to orient and the 
number of apples correctly oriented for the inverted cone — wheel 
device using four apple cultivars (Figure 1). This device leaves 
the stem/calyx axis vertical, requiring some form of a spindling 
mechanism to lift the apple up out of the cup and rotate the 
fruit for viewing with a camera. This method of fruit handling in 
a conveying stream of fruit moving at a rate of 6 - 10 apples per 
second was impossible without causing further damage to th 
apples. i, 


OBJECTIVES 


a Determine if apples can be oriented using two commercial 
bicone conveying systems and two experimental conveying systems. 
Phe If the apples are oriented, determine how long the apples 
will remain oriented. 

ou Evaluate the effect of apple height and width on the ability 
of each conveying device to orient apples. 


MATERIALS AND METHODS 


Two groups of apples were used for testing. Group 1 
contained 50 apples each of Empires, Cortlands, and Red 
Delicious. One month later, group 2 containing 50 apples each of 
Empires, Jonagold, Ida Red, and Red Delicious was selected from 
storage. The apples were harvested during the fall of 1994 and 
held in cold storage until 24 hours before testing. Each apple 
waS numbered and height and width were recorded. By selecting 
group 2 one month later than group 1, two 50 apple sets of 
Empires and Red Delicious were available for testing which likely 
came from different trees. This makes it more likely that the 
affect of tree to tree variation in apple height and width on 
orientation is included for these two cultivars. 

Each apple from group 1 was oriented by methods 1, 2, and 3. 
Each apple in group 2 was oriented by methods 1, 2, 3, and 4. 


Method 1: Two plastic bicones and a cylindrical padded roller 
were assembled as shown in figure 2. The cylindrical padded 
roller rotated 96 rpm and the bicones were free to rotate 
independently. 


Method 2: The two plastic bicones were assembled different from 
method 1 as shown in figure 3. The cylindrical padded roller 
rotated 96 rpm and the bicones were free to rotate 
independently. 


Method 3: A commercial bicone system using four rotating molded 
rubber bicones turning at 96 rpm to rotate fruit while being 
conveyed (Figure 4). 


Method 4: A different commercial bicone system using bicones 
slightly smaller in diameter with curved surfaces turning at 120 
fom (Figure 5). 


All of the conveying assemblies were mounted on a custom 
laboratory device with drive belts driven by an adjustable speed 
drive so that the appropiate bicones or rollers were rotated 
during testing. The conveyor segments did not convey the fruit 
but were held in a stationary position while an average sized 
apple (70 to 75 mm diameter) rotated at speeds typical to the 60 
rpm rotating speed found on moving conveying systems on 
commercial sorters. 

Apples were placed on each system with the axis passing 
through the stem and calyx ends parallel to the axis of rotation 
of the bicones and rollers. This position was considered to be 
the oriented position. The time that each apple remained in this 
position was recorded as the time that each apple stayed 
oriented. 

Apples were placed on each system with the axis which passes 
through the stem and calyx ends perpendicular to the axis of 
rotation of the bicones and rollers. The time for thls axis to 
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become parallel with the axis of rotation of the rollers and 
bicones was recorded as the time for apple orientation. 

The maximum time allowed for apples to orient or to stay 
oriented was 20 seconds. 

Time was measured with an electronic stop watch (Model 460, 
Accusplit Inc., 2290A Ringwood Ave., San Jose, CA. 95131). The 
height and width of each apple was measured with a digital 
vernier caliper (Model 599-571-3, Brown and Sharpe). 


RESULTS AND DISCUSSION 


Figures 6 - 9 show the percentage of the fifty apples for 
each of 5 cultivars that were correctly oriented using one of the 
four previously described methods at two second intervals of time 
up to a total time of 20 seconds. The percentage of the fifty 
apples for each of 5 cultivars that stayed oriented using one of 
the four previously described methods of orientation is also 
shown at two second time intervals up to a total time of 20 
seconds. All apples tested in Figures 6 - 9 came from group 2 
except Cortlands. Also Cortlands were not tested in method 4 
because they were no longer available from storage when the 
conveying device was available. 

Method 1 oriented Cortlands (100%), Empires (88%), Idareds 
(96%) and Jonagolds (98%). If orientation occurred, it was 
sucessful after 10 - 13 seconds. Furthermore, once oriented, they 
stayed oriented at least 98% percent of the time for 20 seconds. 
However, method 1 was unsuccessful in orienting Red Delicious 
(12%) and only 60% stayed oriented for the 20 seconds (Figure 6). 

Method 2 oriented Cortlands (83%), Empires (73%), Idareds 
(90%) and Jonagolds (95%) after a period of 16 - 18 seconds 
compared to 10% of the Red Delicious. Only Cortlands (96%) stayed 
oriented. After a period of 10 seconds, only 82% of the 
Jonagolds, 72% of the Idareds, 58% of the Empires, and 0% of the 
Red Delicous remained oriented (Figure 7). This method required 4 
- 5 seconds longer to orient apples. As the oriented fruit 
rotated, they tended to wobble slightly about its axis of 
rotation caused by small geometric irregularities of the apple’s 
shape. With time this motion caused the oriented apple’s axis of 
rotation to become unstable resulting in fruit disorientation. 

Method 3 and Method 4 could not orient Empires, Cortlands, 
Idareds, or Jonagolds. In fact, the axis passing through the stem 
and calyx ends consistently changed to perpendicular to the axis 
of rotation of the bicone rollers for the 4 cultivars. After 16 
seconds, method 3 oriented 68% and method 4 oriented 60% of the 
Red Delicious. After a 14 second period, less than 30% of the 
Empires and less than 20% of Cortlands, Idareds, or Jonagolds 
stayed oriented. After 16 seconds, 70% of Red Delicious stayed 
oriented for method 3 compared to 65% for method 4 (Figures 8 and 
oe 

Orientation appears to be dependent on both cultivar and 
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method for achieving orientation. In fact, orientation can be 
dependent on apple shape factors even within a single cultivar. 
Red Delicious oriented better using methods 3 and 4 but with a 
30% to 40% failure. The other 4 cultivars oriented best with 
method 1 with a 0% to 12% failure. Figure 10 compares 2 different 
50 apple samples, one from group 1 and one from group 2, for the 
cultivars of Empires and Red Delicious using method 1. The second 
set of Empires (97%) oriented better than the first Empire sample 
(88%). The first set from group 1 of Red Delicious (60%) 
oriented better than the second set of Red Delicious (12%) from 
group 2. Except for only 60% of the second set of Red Delicious 
from group 2, 95% of group 1 Red Delicious and both eounS of 
Empires stayed oriented for 20 seconds. 

Figure 11 shows plots of apple height and eoete Width for 
both 50 Red Delicious apples from group 1 (top) and 2 (bottom) 
versus time to orient in seconds with all data points at 20 
seconds representing apples which did not orient. It appears that 
apples that oriented had widths greater than their heights. 
However, at least some of the apples that matched this criteria 
did not orient, indicating that other features such as surface 
shape or curvature and apple size may be variables, as well as 
method, which determine if apples orient. The Red Delicious 
apples from group 2 have larger heights and widths than Red 
Delicious apples from group 1. The relationship between the 
spacing between the cones and also between the cones and the 
cylindrical roller to apple size may have been a factor which 
caused poorer orientation of group 2 Red Delicious. However, 
shape, curvature, or some other features are involved since 
Cortlands had larger heights and widths (Figure 12) and still 
oriented. Figures 12 and 13 show apple height and width versus 
time to orient for Empires, Cortlands, Idareds, and Jonagold 
using method 1. All of these cultivars had widths greater than 
their heights. Empires, which had the poorest record of 
successful orientation (88%), had the smallest difference between 
the height and width for each fruit. Another factor that came 
into play with Empire orientation is their long stiff stems which 
interfered with the apple rotating into the oriented position. 
Other cultivars tested did not have long stems. 

Observation of the 4 methods of orientation found that 
methods 1 and 2 have three points of fruit contact compared to 
four points of contact for methods 3 and 4. It appears that the 
apples seek a position upon the contact points where the two 
rings of contact points about the fruit have equal circumferences 
and where the axis of rotation for the fruit lies through 
Shortest dimension of the fruit, in this case the apple height. 
In methods 1 and 2, this is impossible to achieve equal 
circumferences with the wider distance (38 - 40 mm) between the 
cone contact points for Red Delicious apples which have the 
elongated pear shape, forming much different fruit diameters at 
the stem end compared to the calyx end, particularly when the 
axis of rotation of the fruit lies through the stem/calyx axis. 
In Northeast United States, Red Delicious have both fruit that 
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are elongated pear shaped and nearly round. The elongated Red 
Delicious can only achieve equal circumferences about the fruit 
when the stem/calyx axis is perpendicular to the axis of rotation 
of the cylindrical roller. This is the position which the 
unoriented fruit stayed for methods 1 and 2. The more round Red 
Delicious without the pear shape orient much like the other 4 
cultivars which have a more symetrical shape. Method 1 oriented 
all of the cultivars except Red Delicious quicker and wath e 
higher rate of success because, by turning the one cone 180 to 
its position in method 2, its contact point formed an edge which 
could aline itself with the fruit edge at the stem/calx ends— 
providing stability to the revolving fruit once oriented. This 
did not work for the elongated Red Delicious when the calyx end 
was resting on this cone because the diameter of the fruit edge 
is very small compared the other 4 more symetrical cultivars. 

Methods 3 and 4 had four contact points, one on each cone. 
The contact points were 25mm apart, narrower than for methods 1 
and 2. With all 4 contact points powered and close together, 
apples tended to position themselves'so that the fruit axis of 
rotation passed through the largest dimension of the apple, the 
apple width. For the Red Delicious elongated pear shaped fruit, 
this did not hold true. The stem end of this fruit has more mass 
than the calyx end, causing the fruit, while on the narrow spaced 
contact points, to topple over when in the position of the 
stem/calyx axis perpendicular to the axis of rotation of the 
bicones. Once toppled, the fruit shifted along its stem/calx 
axis, now the axis of rotation, with the calyx end moving further 
away from the contact points until rings of equal circumference 
are found. This motion can actually cause the calyx end of large 
elongated Red Delicious to extend outside of the bicones and 
interfere with fruit in the next lane. 

Further studies need to consider apple shape and curvature 
and determine how these factors relate to width between cones and 
the width between the bicone rollers and the cylindrical roller. 
The angle and shape of the cone surface could also be a factor. 


CONCLUSIONS 


Of the four methods tested, method 1 correctly oriented the 
stem/calyx axis into the position of the axis of rotation of 
Cortland, Jonagold, Idared, and Empire apples correctly (88%- 
100%) in the shortest amount of time (12 seconds). For two 50 
apple sets of Red Delicious, 62% and 12% were correctly oriented. 
The elongated pear shape of this cultivar determined if 
orientation occurred for all methods tested. 

A difference between apple height and apple width plays a 
part in the orientation processes tested, but other factors such 
as fruit shape, size, stem length, stem stiffness and surface 
Curvature appear to play a roll. 

Present commercial bicone conveyors (methods 3 and 4) 


oriented 68% and 60% of the Red Delicious respectively. All fruit 


of the cultivars of Cortland (not tested by method 4), Idared, 
Jonagold, and Empires assumed a position with the axis of 


rotation of 
marough the 
orient also 
orientation 


orientation. 


the apple perpendicular to the axis of a line passing 
stem/calyx ends. The Red Delicious which failed to 

assumed this position. It appears that the commercial 
methods depend on an elongated pear shaped apple for. 
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Figure 1. Time to orient and the percentage of correctly oriented 
fruit using the inverted cone - wheel device for four 
apple varieties (Rehkugler and Throop, 1986). 
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Figure 2. Picture of the experimental hardware used in method i 
for apple orientation consisting of two freely rotating 
cone rollers and a powered cylindrical padded roller. 





Figure3. Picture of the experimental hardware used in method 2 
for apple orientation consisting of two freely rotating 
cone rollers, one rotated 180° from its position for 
method 1, and a powered cylindrical padded roller. 





Figure 4. Picture of a commercial bicone conveyor used in method 


3 for apple orientation. The bicones were rotated at 


normal grading machine rate but not used to convey 
fruit during testing. 





Figure 5. Picture of a commercial bicone conveyor used in method 
3 for apple orientation. The bicones were rotated at 
normal grading machine rate but not used tod convey 
fruit during testing. 
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Figure 7. The percent of correctly oriented apples (top) versus 
time to achieve orientation and the percent of apples 
which remained oriented (bottom) versus time stayed 
oriented using method 2 is shown for 5 cultivars. 
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Figure 8. The percent of correctly oriented apples (top) versus 
time to achieve orientation and the percent of apples 
which remained oriented (bottom) versus time stayed 
oriented using method 3 is shown for 5 cultivars. 
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The percent of correctly oriented apples (top) versus 
time to achieve orientation and the percent of apples 
which remained oriented (bottom) versus time stayed 
oriented using method 4 is shown for 4 cultivars. 
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Figure 10.The percent of correctly oriented apples (top) versus 
time to achieve orientation and the percent of apples 
which remained oriented (bottom) versus time remained 
oriented using method 1 is shown for four 50 apple 
samples, two of Empires and two of Red Delicious. One 
month difference in storage time between sample pairs 


for each cultivar assured apples from different trees. 
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Figure 11.Apple height and width versus time to orient using 
method 1 for two 50 Red Delicious apple samples 
selected one month apart from storage. All heights and 
widths at a time of 20 seconds represent apples which 


did not orient. 18 : . Peo eel da 
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Figure 12.Apple height and width versus time to orient using 
Seer Lor Cortlands (top) and Empires (bottom). All 
eights and widths at a time of 20 seconds represent 
apples which did not orient. 
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Figure 13.Apple height and width versus time to orient using 
method 1 for Idareds. (top) and Jonagold (bottom). All 
heights and widths at a-time of 20 seconds represent 
apples which did not orient. 
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INTRODUCTION 


Considerable effort has been applied to sensor techniques to nondestructively sense the internal 
and surface quality of apples. Two trains of thought regarding the handling of the fruit during 
this process have been taken: 1) Convey the fruit past the sensor on some form of rotating 
cones or rollers while using the sensor to locate both the stem/calyx regions and any defect, or 
2) Orient the fruit in such a manner that the locations of the stem/calyx regions can be predicted 
and use the sensor to locate only the defects. 


Structured light has been used to differentiate the concave surfaces around the stem/calyx ends 
from blemishes on the rest of the convex apple surface (Yang, 1992). The identification 
process has been carried a step further by applying neural networks to identify the altered 
structure of the light (Yang, 1993). The fruit packing industry requests that apples must be 
sorted at rates of 6 - 10 apples per second. There is little processing time to waste on locating 
the stem/calyx ends. If the stem/calyx ends pass directly through the camera view, extra 
lighting and mirrors or cameras must be added to view the sides of the apple as it rotates on 
bicone rollers or else this portion of the apple cheek will not be inspected. The stem/calyx could 
also be mistakenly identified as a defect. 


An automatic detection system for surface defects on apples using a commercial sorting 
conveying system of bicone rollers used an algorithm dependent on the fruit rotating about a 
horizontal axis running through the stem and calyx cavities. During their study, video 
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recordings of commercial sorters equipped with "bicone" conveyors showed that fruit did not 
rotate as their algorithm required (Davenel et. al., 1988). 


Many researchers have attempted to measure apple firmness by different techniques all 
requiring knowledge of where the stem/calyx regions are located. Orientation or stem/calyx 
identification is a requirement if this type of measurement is considered (Abbott, 1994), 
(Armstrong and Brown, 1991), (Pitts et. al., 1991), (Peleg, 1993), and (Chen et. al., 1993). 


Two apple orientation devices that place the stem/calyx regions in a known location have been 
built in the past (Rehkugler and Throop, 1976). One device consisted of a rotating inverted 
conical shaped transport cup with an opening at the bottom . Through the opening a small 
rotating wheel contacted the fruit until the surface indentation of the stem/calyx was positioned 
over the wheel causing loss of apple surface contact with the wheel ending apple rotation 
(Keesling, 1965). The time to orient and the number of apples correctly oriented for the 
inverted cone - wheel device using four apple cultivars was recorded (Figure 1). 





Time in Seconds 


—=s- Mcintosh-— Cortland—@— Crispin —s~— Empire 


Figure 1. Time to orient and the percentage of correctly oriented fruit using the inverted cone - 
wheel device for four apple varieties (Rehkugler and Throop, 1986). 


This device leaves the stem/calyx axis vertical, requiring some form of a spindling mechanism 
to lift the apple up out of the cup and to rotate the fruit for viewing the whole apple surface 
with a camera. A conveying stream of fruit moving at a rate of 6 - 10 apples per second was 
impossible by this device without causing further damage to the fruit. 


A second device consisted of a combination of free-to-rotate opposing cone rollers on one shaft 
and a powered rotating padded cylindrical roller on a second shaft. After orientation to an axis 
of rotation about an axis passing through the stem/calyx ends, some fruit tended to wobble. One 
of the cones was changed to a vertical disk which tended to engage the shoulder formed 
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between the concave and convex surfaces that intersect at the stem/calyx ends resulting in less 
wobble. This device correctly oriented 88% - 100% of Empires, Cortlands, Jonagolds, and 
Idareds (Throop et. al., 1995). The fruit was not grouped by size for this study. 


Further modifications were made to the cone shaped roller changing the flat angular surface to a 
concave surface. The vertical disk and cone shaped roller were spring loaded towards each 
other allowing lateral movement along the common shaft about which they rotate. This allowed 
a greater range of fruit sizes to be oriented. Testing was carried out with the following 
objectives in mind. 


OBJECTIVES 


1. Determine which apple cultivars can be oriented using two commercial bicone conveying 
systems and one experimental conveying system. 

2. If the apples are oriented, determine how long the orientation process takes and how long 
the apples will remain oriented. 

3. Determine if the new device can handle a range of large and small fruit without altering the 
geometry between the disk/cone and cylindrical roller. 


MATERIALS AND METHODS 


Apples were harvested and placed in cold storage during the Fall of 1995. They were removed 
from storage during November 1995 and sized into two groups, 88's and 128's (88 and 128 
apples per standard container consisting of 4 tray packs) before testing (Omni Sort, Durland- 
Wayland, Lagrange , GA). The typical sample size was 44 apples except when to few apples of 
a size catergory for a particular cultivar could be found (Table 1). 


Three conveying and orienting devices were tested. 


Experimental Device: Consisted of a plastic bicone with a concave surface and a flat vertical 
disk assembled on one shaft parallel to a cylindrical padded roller on a second shaft (Figure 2). 
Both the flat disk and bicone were spring loaded with compression springs toward each other 
allowing them to move horizontally along their common shaft compensating for different sized 
fruit. The cylindrical padded roller rotated 96 rpm producing an apple rotational speed of about 
60 RPM. The bicone and disk rotated independently of each other and dependently on their 
contact with the fruit surface. 
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Table 1. The number of apples and size for each cultivar tested for orientation. 


Cultivar Size 128 Size 88 
Golden Delicious 44 44 
McIntosh 10 44 
Fuji 41 44 
Rome 44 43 
Gala 5 44 
Jonagold 14 44 
Crispin 25 44 
Cortland 22 44 
Empires a 18 
Granny Smith 22 44 
Red Delicious (East) 44 44 
Red Delicious (Wash) 44 44 





Figure 2. Picture of experimental conveying device after apple orientation. 


Commercial Device 1: A commercial bicone system consisting of two pairs of powered 
molded rubber bicones rotating on two separate parallel shafts with their angular flat surfaces 
facing towards each other turning at 96 rpm. Contact with the apple surface produced apple 
rotation of about 60 RPM (Figure 3). 
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Commercial Device 2: A commercial bicone system consisting of two pairs of powered molded 
rubber bicones slightly smaller in diameter rotating on two separate parallel shafts with their 
angular concave surfaces facing towards each other turning at 120 rpm. Contact with the apple 
surface produced apple rotation of about 60 RPM (Figure 4). 





Figure 3. Picture of a commercial bicone conveyor after apple orientation. 





Figure 4. Picture of a second commercial bicone conveying device after orientation. 
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The conveying assemblies were mounted on a custom laboratory device with drive belts driven 
by an adjustable speed drive providing power to the appropiate bicones or rollers during 
testing. The conveyor segments did not convey the fruit but were held in a stationary position. 
Contact with the surface of an average sized apple (70 to 75 mm diameter) resulted in rotational 
speeds of about 60 rpm, typical of the apple rotational speed of moving conveying systems on 
commercial sorters. 


The first experiment, placed apples on each system so that the axis passing through the 
stem/calyx ends was parallel to the axis of rotation of the bicones and rollers. This position was 
considered to be the oriented position. The time that each apple remained in this position was 
recorded as the time that each apple stayed oriented. 


In the second experiment apples were placed on each system so that the axis which passes 
through the stem/calyx ends was perpendicular to the axis of rotation of the bicones and rollers. 
The time for the stem/calyx axis to become parallel with the axis of rotation of the rollers and 
bicones was recorded as the time for apple orientation. 


The maximum time allowed for apples to orient or to stay oriented was 0.3 minutes. Time was 
measured with an electronic stop watch (Model 460, Accusplit Inc., 2290A Ringwood Ave., 
San Jose, CA. 95131). 


RESULTS AND DISCUSSION 


Condition 1: Apples placed on experimental conveying device (Experimental Device) with the 
axis of rotation perpendicular to the stem/calyx axis. 


After 0.2 minutes, Cortland, Empire, Fuji, Jonagold, McIntosh, and Rome apples oriented so 
that their axis of rotation coincided with an axis passing through the stem and calyx with a 95% 
to 100% accuracy. The exception was the large Fuji which oriented only 82% correctly 
(Figure 5). The same cultivars remained oriented longer than 0.2 minutes 97% to 100% of the 
time when they were placed on the device so that their stem/calyx was parallel to the axis of 
rotation of the cone/disk and the roller. If the apple rotational speed for the same 6 cultivars 
was increased from 60 RPM to a rotational speed of 90 RPM, some of the fruit oriented 
correctly in a shorter length of time as indicated by a greater slope of curves between times 0 
minutes and 0.05 minutes. However, the apples that required the most time to orient for 60 
RPM required more time to orient for 90 RPM. This reduced successful orientation for small 
Empires from 94% to only 83%, large Jonagolds from 97% to 87% , and increased successful 
orientation of large Fujis from 82% to 88% after a time of 0.2 minutes (Figure 6). 
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Experimental Device - Size 128 - Apple Rotation 60 RPM 
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Figure 5. Percent of apples rotating at 60 RPM of size 128 (top) and size 88 (bottom) which 
changed their axis of rotation from perpendicular to the stem/calyx axis to the same as the 
stem/calyx axis using the experimental device. 
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Experimental Device - Size 128 - Apple Rotation 90 RPM 
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Experimental Device - Size 88 - Apple Rotation 90 RPM 

100 
80 
60 
40 
20 


Percent Oriented 





Time in Minutes 


—=— Cortland —»— Empires —=— Fuji 
—— Jonagold —=— Mcintosh —=- Rome 


Figure 6. Percent of correctly oriented apples using an apple rotational speed of 90 RPM for 
size 128 (top) and size 88 (bottom) apples. The axis of rotation was changed from perpendicular 
to the stem.calyx axis to about the stem/calyx axis using the experimental device. 


Golden Delicious, Granny Smith, Crispin, Red Delicious, and Gala (unable to find 88 size) 
have a greater variation in overall shape causing difficulty in orientation by any device. In 
some cases the apple shape approaches that of a sphere (height and width equal) which has no 
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geometrical bias favoring a specific orientation. With the exception of large Granny Smith 
(57%) and large Crispin (52%), successful orientation to a rotational axis about the stem/calyx 
axis was achieved 60% to 82% of the time while using the experimental conveying device 
(Figure 7). Controlling the maximum size as well as the minimum size for the 88 sized fruit 
would improve the success rate of the large Granny Smith and Crispin cultivars as some of the 


Experimental Device - Size 128 - Apple Rotation 60 RPM 
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Experimental Device - Size 88 - Apple Rotation 60 RPM 
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Figure 7. Percent of apples rotating at 60 RPM of size 128 (top) and size 88 (bottom) which 
oriented from an axis of rotation perpendicular to the stem/calyx axis to an axis of rotation 
about the stem/calyx axis using the experimental device. 
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fruit were just to large for the conveying device. If the 5 cultivars were placed on the device in 
an oriented position, 70% or more of the fruit stayed oriented in this position of rotating about 
the stem/calx axis for 0.2 minutes. 


Commercial Device 1 - Size 128 - Apple Rotation 60 RPM 
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Commercial Device 1 - Size 88 - Apple Rotation 60 RPM 
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Figure 8. Percent of apples of size 128 (top) and size 88 (bottom) which continued rotating 
about the stem/calyx axis using a commercial bicone conveying device described in method 2. 
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Condition 2: Apples placed on a commercial bicone conveying device (Commercial Device 1) 
with the axis of rotation about the stem/calyx axis. 


When Cortland, Empire, Fuji, Jonagold, McIntosh and Rome apples were placed on the 
commercial conveying device with their axis of rotation about the stem/calyx axis, 60% of the 


Commercial Device 1 - Size 128 - Apple Rotation 60 RPM 
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Commercial Device 1 - Size 88 - Apple Rotation 60 RPM 
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Figure 9. Percent of apples of size 128 (top) and size 88 (bottom) which continued rotating 
about the stem/calyx axis while using a commercial device 1. 
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fruit regardless of size assumed a new axis of rotation perpendicular to the stem/calyx axis after 
a period of time of 0.10 minutes (Figure 8). After 0.20 minutes, 80% or more of the fruit had 
made this axis of rotation shift. For this particular axis of rotation, the stem/calyx ends would 
rotate directly through the center of the field of view of an overhead camera. A large portion 
of the surface area composing of the apple cheek would not be in the camera field of view and 
would not be inspected. This orientation response was typical for these 6 cultivars regardless of 
the commercial bicone conveying device used, commercial device 1 or 2. 


After a period of 0.10 minutes, 40% or more of Golden Delicious, Granny Smith, Crispin, Red 
Delicious, and Gala changed from an axis of rotation about the stem/calyx to one of rotating 
about an axis perpendicular to the stem/calyx axis. This increased to 65% after 0.20 minutes 
(Figure 9). Many of the apples of these 5 cultivars approached the shape of a sphere (height and 
width equal) which has no geometrical bias favoring a specific orientation thus fewer apples 
compared to the other 6 cultivars tested changed orientation but still a significant number did 
shift axis of rotation to that of perpendicular to the stem/calyx axis. Similar results were 
obtained for both commercial device 1 and commercial device 2. 


Condition 3: Red Delicious apples from Washington State placed on the experimental device 
rotating about the stem/calyx axis and placed on the bicone conveyors (commercial devices | 
and 2) rotating about an axis perpendicular to the stem/calyx axis. 


It was observed that the bicone conveying devices tended to orient apples which had a height 
(distance between the stem and calyx ends) greater than the maximum diameter measured 
measured perpendicular to the stem/calyx axis. Only a few apples from the 11 cultivars grown 
in the eastern United States grew with this characteristic shape. Red Delicious apples grown in 
the State of Washington typically grow with this characteristic elongated shape with a height 
greater than their maximum diameter. Apples of the 88 and 128 size were obtained from the 
State of Washington and tested. Regardless of size, most of the apples changed from an axis of 
rotation about the stem/calyx axis to an axis of rotation perpendicular to the stem/calyx axis 
when using the experimental device (Figure 10). 


The commercial devices 1 and 2 oriented 88% to 96% of the fruit from an axis of rotation 
perpendicular to the stem/calyx axis to an axis of rotation about the stem/calyx axis (Figure 10). 
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Size 128 - Apple Rotation 60 RPM 
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Figure 10. The percentage of Red Delicious apples harvested from Washington State of size 
128 (top) and size 88 (bottom) that stayed oriented using the experimental conveying device and 
correctly changed orientation to rotate about the stem/calyx axis while using bicone roller 
conveyors (commercial devices 1 and 2). 
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CONCLUSIONS 


It is clear that the commercial bicone roller conveyors (commercial devices 1 and 2) presently 
used on sorting machines today orient primarily one shape, that of an apple elongated so that 
the height measured as the distance between the stem/calyx ends is greater than a maximum 
diameter measured perpendicular to this height. On the commercial bicone rollers, Washington 
State Red Delicious apples oriented to a rotational axis about the stem/calyx axis with a 88% to 
96% success rate. When 11 other cultivars grown in Eastern United States were placed on this 
device, 60% to 100% of the fruit assumed an axis of rotation about an axis perpendicular to the 
stem/calyx axis. When considering automatic fruit inspection for these 11 cultivars, this places 
a large portion of the surface area composing of the apple cheek out of an overhead camera's 
field of view and these hidden areas would not be inspected. 


The experimental device oriented 95% to 100% of Cortland, Empire, Jonagold, McIntosh, 
Rome, small Fuji apples and 82% of large Fuji apples so that fruit rotation was about the 
stem/calyx axis. With the exception of large Granny Smith (57%) and large Crispin (52%), 
successful orientation was achieved 60% to 82% of the time for Golden Delicious, small 
Granny Smith, small Crispin, Red Delicious, and small Gala apples. All of the Washington 
State Red Delicious apples assumed an axis of rotation perpendicular to the stem/calyx. 


The lateral movement of the disk and angular concave surfaced cone for the experimental 
device allowed comparitive success of orientation of both fruit sizes. With the exception of 
very large Granny Smith and Crispin (reduced 5% to 10 %), little difference was found for 
successful orientation between fruit sizes for most cultivars. 
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Summary: 


A computer vision algorithm was developed to identify stem and calyx of rotating apples. The algorithm 
uses structured light for the identification. Representative samples of all possible orientations of the apple 
where considered in the evaluation of the algorithm. The algorithm considers implementation problems 
such as the elimination of background from the image. Results show that structured lighting is a feasible 


solution to detecting the stem and calyx of apples. Further research is required to optimize some 
parameters of the algorithm. 
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Abstract: 
An algorithm that uses three-dimensional information of the apple skin was developed to identify stem and 
_ calyx of apples. A structured lighting system was used to obtain the three dimensional information of the 
apple surface. The setup used to capture image data was a replica of the one that would be used in an 
industrial environment. The location of the stem and calyx were estimated by analyzing the geometry of the 
reflected light pattern. The parameters used to estimate the location of the stem and calyx were the rate of 
change in the slope of the projection lines and the pattern of the discontinuities of the light patterns. The 
analysis of the light pattern was based on the assumption of spherical apples. A sampling methodology was 
implemented to ensure that representative samples of all possible apple orientations were used. The 
algorithm was evaluated based on the difference between the actual location of the calyx/stem and the 
algorithm’s estimate of location. The apple variety used for the algorithm evaluation was Red Delicious. 
This variety has very irregular geometry and presented a worst case scenario for the algorithm evaluation. 
The results obtained indicate that the algorithm used is adequate to locate the stem and calyx of the apples, 
especially when the stem/calyx are located near the center of the apple silhouette. Further research needs 
to be done to observe the effect of the setup geometry on the algorithm performance. It was also 
recommended to investigate the possibility of detecting the calyx/stem using a pattern classifier, since this 
would simplify the overall grading system. 


Objectives: 

The objective of this work was to identify a feasible computer vision solution to identify the stem and calyx 
of Red Delicious apples when they appear in the field of view of the camera for the possible orientations at 
which the stem and calyx could be observed by the camera used for inspection. Develop the image 
enhancements required to successfully implement the technique used. Analyze the expected error in 
estimating stem or calyx location at each possible orientation of the apple. Examine other possible 
techniques that could be used to identify the stem and calyx of the apples. Suggest future work that should 
_ be done including considerations of other techniques for identifying the stem and calyx as well as 
optimizations for the algorithm developed. 


Introduction: 

Currently one of the limitations in implementing an automated apple grading system is that it is not possibly 
to mechanically orient all apples (Throop et al., 1997). The knowledge of apple orientation during the 
inspection process has been recognized to be important for automated grading (Rehukugler et al., 1986) 
(Throop et al., 1988). Throop et al. (1997) compared the performance of two orienting devices. The goal of 
the orienting device was to rotate all apple varieties along the stem/calyx axes. This would expose the 
cheek of the apple for inspection and avoid the appearance of the stem and calyx in the field of view of the 
camera. The results showed that the varieties that were successfully oriented with one system would not 
orient using the other device. 


Another problem related to the uncertainty of the apple orientation is that the stem and calyx could be 
misclassified as blemishes (Yang, 1993) (Throop, 1997). One of the approaches followed in the research is 
to use the spectral reflectance from the apple surface at various wavelengths. Aneshanley et al. (1997) 
completed a reflectance spectra analysis of several surface defects on various apple varieties. The study 
concluded that some other features besides reflectance information would be needed to identify all possible 
defects present on the apple’s surface. In addition, it seems not possible to use reflectance information to 
distinguish the stem and calyx of the apple from skin defects (Personal Communication. Throop, 1997). 
This will lead to erroneous classification of all apples not oriented at time of evaluation by the system. 


Yang (1993 and 1996) used structured lighting to identify the stem and calyx of apples. The method used 

_ two images: one image using diffuse lighting and a second image using the structured light source. The 
criteria to distinguish the stem and/or calyx from other defects was based on curvature of the laser lines, 

and this parameter is heavily dependent on the apples being spherical. This model is accurate for most 
varieties, but it is clearly not an accurate representation of varieties such as Red Delicious. No information 
is provided regarding the conditions on which the algorithm fails to identify the stem and calyx. In addition, 
the procedure to ensure that all angles of view were covered and the effect of orientation were not clear 
from either of Yang's papers. The study presented here, implements an algorithm similar to that developed 
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by Yang (1993) and examines the performance of the technique at all possible locations of the stem and 
calyx, as well as the effect of using non-spherical apples. The algorithm developed in this work also 
considers the pattern of broken line projections in the image. 


System Setup Overview: 
The stem and calyx identification was achieved by means of a structured lighting setup. The structured light 
source was a 1-mW diode laser (SNF-533L-685-20-5-038. Lasiris, Inc.). This laser was provided with lenses 
that produced a pattern of parallel lines. The fan angle of the laser stripes was 5° and the interbeam angle 
was 0.38°. The maximum number of laser stripes that could be projected onto the apple was 7 to 9 lines, 
depending on the size and orientation of the fruit. The lines were projected parallel to the axis of rotation 
(see Figure 1). The angle between the focal camera axis and the laser light beam was set to 20°. The 
projections of the laser light into the scene were recorded using a BAW CCD solid state camera (model: 
4815-5000/0000. Cohu). The lens used during the experiment had a focal distance of 15mm. The 
resolution of the image was set to 256x256 pixels, and the area of the field of view was set to a xD) 
(12.7x12.7 cm). The frame grabber used to digitize the image was a Raptor-1M (RAP-PCI-VS-1, BitFlow, 
Inc., Woburn, MA). The software used to capture the images was Quantim (Zedec Technologies, Inc., 
Burlington, NC). 


Structured Light 


| Source 
| 
Camera \ Rw! 
| 
205 
| 
Apple 
Conveyor 
4 elt 





Figure 1. A Red Delicious apple with eight line of structured light parallel to the axis of rotation. 


This work was to determine the error associated with the estimation of the calyx and stem location, on the 
image, at each possible orientation of the apple. To achieve this, the apples were manually oriented on the 
still belt to ensure that all possible orientations of the apple were considered in the analysis. The 
consideration of all possible apple orientations was achieved by superimposing a grid onto a monitor while 
viewing the apple with the camera. The calyx and the stem were manually positioned at the center of each 
square of the grid. The grid had 64 squares, each of which was 32x32 pixels and covered an area of 
0.625"x0.625” (1.6cmx1.6cm). All feasible orientations were considered for both the stem and calyx. Fora 
large size Red Delicious apple, this technique yielded approximately 40 to 55 orientations. 


The evaluation of the algorithm required the acquisition of three images for each possible apple orientation: 
no ambient light, some ambient light, and saturated apple silhouette. The image with some ambient lighting 
was used for display purposes and for verification of calyx and stem location. This image was captured by 
using a 60-W incandescent lamp located above the camera. The saturated apple silhouette was formed by 
illuminating the scene with incandescent light from the sides of the image. The positioning of the lamp for 
ambient and saturated conditions was not critical. The laser light source was turned off during the 
positioning of the apple to ensure that there was no bias in the final setup of the image. The image with ’ 
background light was used to display the results and to ensure that the calyx was positioned correctly. The 
image with no background light was used to analyze the laser light pattern and to estimate the location of 
the calyx or stem. The last image, the saturated apple silhouette, was required to find the center of mass of 






































| image, while the rest of the image was set to zero. The thinning was achieved by detecting the maximum 
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the apple and to normalize the apple size. All three images were taken without relocating the apple, 


ensuring that the images had a perfect pixel to pixel correspondence. The operation of the algorithm only 
requires the image with no ambient light. 


The algorithm was developed and tested using images from three different groups of apples. Images from 
the first group were captured using laser line projections and no background light. This set of images used 
four apples and a total of over 160 images, slightly over 40 orientations per apple. These images were used 
only for initial developing and debugging of the algorithm. A second set of images was taken using two 
different apples. The three image setups described above were taken for each of the possible orientations 
of the apples. These images were used to train the algorithm. A third set of images was used to evaluate 
the performance of the algorithm. The performance of the algorithm was measured as the Euclidean 
distance from the actual to the estimated location of the stem or calyx. All three images were taken for 
each of the apple orientations. Seven apples were used in this test, yielding over 350 views (1050 images). 


Algorithm Description: 


The algorithm developed for the stem and calyx identification consists of image preprocessing and 
hypothesis forming. The preprocessing part of the algorithm reduces the original image to a form of 
information that can be readily used to estimate the location of the stem and calyx. The preprocessing 
included: structured light thresholding, thinning, labeling, background projection removal, contour 
smoothing, determining slope and rate of slope change and connecting line segments. The hypothesis 
forming used the values of the slope, rate of change in slope and information about the connectivity of the 
line segments. The algorithm is shown schematically in Figure 2. 


Connect Contour | Contour segments are 
Segments connected based on 
distance and compatibility 
Check for 
Calyx/Stem 









The image is thresholded 
line by line using output 
from edge detector 


Threshold Image 
(modified edge detection) 
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Figure 2. Schematic Representation of the Algorithm 


Identification and Thinning of the Laser Line Projections: 

The first goal of the algorithm is to identify all the pixels that correspond to laser projections. The laser line 
projections.are identified using a.modified Sobel edge detector (Jain et al., 1995). The filter mask used was 
defined by the convolution of a 3x1 step edge and a 3x1 mean filter. The filter mask resulting from the 
convolution was [-1-1 0 1 1]. There was no need to use a mask larger than a row vector because most of 


_ the edge information is in the vertical axis of the image (laser lines are projected parallel to the vertical axis 
_ Of the image, see Figure 1). This size of filter proved to be adequate for the profile of the lines to be 
| identified. A structured line projection was defined as the region of the original image between a local 


maximum and minimum response of the edge detectors. These regions were left unchanged on the original 


gray value inside each of the unchanged regions of the image along each horizontal line of the image. 
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Labeling: 

It is impenent to identify the laser line to which each pixel belongs. This simplifies the implementation of 
the following sections of the algorithm. Two pixels were considered to belong to the same contour, if they 
were within a 5x5 pixel window. This is necessary because the output from the contour detection section 
often yields broken line segments. This is due to the fact that, the algorithm used to perform the thinning is 
restricted to identify one pixel per horizontal line of the image. Therefore, the smallest slope that a line can 
have without being broken is plus or minus one (445°). 


Removing Background: 

The method used to determine which pixels to erase is shown graphically in Figure 3. Figure 3 shows a 
typical image with three laser lines projecting on the background as well as the apple. It was impossible to 
remove the background lines by means of fitting a simple line. Due to this, it was necessary to use a 
maximum and minimum allowable slope and the origin of the fitted line. The expected origin of the line was 
set to the first and/or last pixel of each line segment, depending on the location of the segment. However, it 
was necessary to have +/- 2 pixels on the origin. The start of the background lines was fixed in the image 
due to the geometry of the conveyor belt. The dimensions of the conveyor belt do not change with time, this 
ensures that the background projections would begin and end at the same points (unless the line projects 
onto the apple). Therefore, only those segments that begin within the areas labeled Search Area in Figure 3 
were considered for removal. The areas are given by the location of the bicone rollers on the belt and the 
width of the laser lines. The area delimited by the dashed lines, in Figure 3, is the region in which the pixels 
belonging to the segment would be erased. Note that only a small number of pixels are lost for those 
segments corresponding to the apple projections. 
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Figure 3. Background Elimination 


Smooth the Contours: 

The contours obtained from the thresholding and thinning algorithm are too noisy due to the imperfections in 
thinning algorithm and to the digitalization of the image. To minimize the effect of this noise, the contours 
were smoothed using a linear mean filter of five pixels length. The smoothing filter was applied along the 
length of each contour. The sum of the x and y coordinates of each group of four pixels was used to 


calculate the slope and the change in slope from point to point (i.e. the first and second derivatives of the 
contours). 


Connect Line Segments after Background Lines Removed: 

Each laser lines projected by the structured light source often yields more than one line segment on the 
image. This is usually due to the calyx or stem appearing on the field of view. However, the contours could 
be broken due to changes in the reflectance properties of the apple surface and/or other abrupt changes in 
the geometry of the fruit. In order to evaluate the source of the discontinuities, it is necessary to identify all 
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the segments that belong to the same laser line projection. The lines were identified based on the number 
of neighbors that each line projection had and the classification of its neighbors. Sometimes there was 
more than one possible (i.e. compatible, based on the above discussion) connection for each of the 
segments. The most likely connection between segments is defined as the connection that minimizes the 
distance between compatible connecting ends of the contours. The operation of connecting line segments 


could be highly simplified using a coding technique for the structured lighting (Jain et al.,1995; Boyer and 
Kak, 1987). 


Calyx/Stem Detection: 

The detection of the calyx/stem was divided into two sections. The first dealt with the possibility of having a 
calyx or stem appearing in the image but not breaking the laser lines. The second case deals with the case 
where the laser lines are broken. The criteria selected to indicate the existence of the calyx or stem on a 
continuous contour was the second derivative of the laser line projections. A point was defined to be a calyx 
if its second derivative was larger or smaller than predefined threshold values. The threshold values were 
determined by trial and error. The threshold values corresponded to the two possible curvatures that can be 
present on any image. A different threshold was used for the maximum and minimum change in slope. 


Negetive Second 
Dervetive 


Expected calyx/steam 
location 
Positive Second 
Denvative 





Figure 4. Graphical Representation of First and Second Derivative 


_ Inthe case of discontinuous segments, it was necessary to fit a line that would fit the model 


| Surface. This is very difficult to do for varieties such as Red Delicious, because their shape is 


very irregular and there is no line model that will always fit the surface, with the information 
given. The best approach was to fit a circumference between connecting ends. The 
circumference was defined by the two ends of the segments and the slope of one of them. A 


_ Calyx/stem was said to occur when the difference between the slopes of the fitted 
_ Circumference and the slope of the segment were different by a predetermined value. This 


approach could not be used when the connecting points were too close. This was because the 
fitted circumference is highly sensitive to the estimate of the location of the connecting points. 
In this case, the likelihood of stem/calyx was determined based on the differences among the 


_ Slopes at points A and B, and the slope of the line connecting the two points. 


' Results and Discussion: 


The algorithm usually yielded more than one estimate of calyx/stem location. It was very 


difficult to take into account each estimate in the evaluation of the algorithm. Therefore, the 
actual location estimated was defined to be the average of all the estimates generated by the 


algorithm. The algorithm performance was evaluated based on the distance between the actual 


location of the stem/calyx and the estimated location provided by the algorithm. 


ASAE Paper 97-3079 Campins, Throop and Aneshansley 


The performance of the algorithm was controlled by many parameters. The optimal value of 
most parameters could be determine by examining the results obtained from a few images, 
since the results were very sensitive to the values of these parameters. This was the case for 
the selection of the threshold for the filter mask output, the window size used to label the 
structured light projections and the number of pixels considered to estimate the slope of the 
line segments. However, there are other parameters that have an impact on the performance 
of the algorithm, whose optimum values are not easily determined. These parameters are the 
thresholds values used by the algorithm to estimate the location of the stem or calyx. The 
optimum value for these constants was estimated by running the algorithm multiple times at 
different settings and manually deciding which combination performed best. The best set of 
parameters was the one that minimized the average error between the actual calyx/stem and 
its predicted location. In addition, it was important to choose the combination of parameters 
that minimized the number of unidentified calyx/stem. 


The results also show that the average error and the number of missed stem/calyx locations do 
not change in the same direction as the values of the parameters are changed. That is, 
changing one parameter often caused an increase in one measure of error anda decrease In 
the other. The accuracy of the algorithm is increased by relaxing the threshold values. This 
means that the system will only detect those features that are obvious to the algorithm. It also 
implies that fewer features would be detected, increasing the number of non-detected stem or 
calyx. Increasing the sensitivity (tightening the threshold values) of the algorithm triggers many 
false detection’s that are due to the changes in skin geometry of the Red Delicious. This leads 
to less accurate estimates on the location of the stem and calyx, but it also ensures that more 
less obvious calyxes/stems are detected. These observations indicate that the selection of the 
parameters should be based on the priority of the algorithm: accuracy or detection ability. 
However, the effect of parameter selection should not be as pronounced as in Red Delicious, 
since the parameters used to estimate the likelihood of stem or calyx are based on a spherical 
model of the apples. 


This set of parameters was used to analyzed images from seven different apples. The apples 
chosen had different geometry’s and sizes. Table 1 summarizes the geometry of the apples 
used. 


Table 1. Apple Geometry’s 


3.30 3.10 lrregular(3 
eae a eee Gad 


*Irregular(1) stands for the most irregular apple and Irregular(5) the most spherical apples. 










The results show that the algorithm yields good results close to the center of the image. This is 
expected, since the image is more sensitive to changes in surface geometry of the apple at the 
center of the image. An unexpected result is that the accuracy of the algorithm seems to be 
largely independent of the radius at which the calyx/stem is located (the radius is the distance 
from the center of gravity of the apple to the location of the stem/calyx). However, the results 
show that with an increase in the radius the number of non-identified calyx/stem increases. 

The local coordinates are calculated with respect to the center of gravity of the apple. The 
center of gravity was determined by thresholding the apple silhouette. Any background on the 
image was removed using morphological opening (Jain et al., 1995). The center of gravity was 
determined using the first order moments of the image (Jain et al., 1995). The radius of the 
apple silhouette was defined as the radius of the circle with the same area as the apple 
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silhouette. Note that this is an underestimate of the radius, since some of the apple views had 
elongated silhouettes. Therefore, normalized errors are slightly overestimated, since the errors 
are normalized with respect to the radius the. The normalized results obtained from the 
algorithm as a function of the distance between the center of gravity and the actual location of 
stem/calyx are shown in Figure 5. A zero error corresponds to non-detected stem/calyx. The 
error magnitude is fairly constant for all locations, except for those where the calyx/stem was 
located close to the outside radius of the apple (when viewed from the top). Close to the 
outside radius, there are significantly more non-detected cases and larger errors. The effect of 
the angular location of the stem or calyx in the image is shown in Figure 6. The angle was 
defined in degrees, with the origin at the center of gravity, positive clockwise, and zero degrees 
in the direction of the laser source. The graph shows no clear trend. It could be argued that 
the algorithm is slightly more accurate in the region close to O degrees. 


Normalized Error vs Distance 


Normalized Error 
(Dist/Radius) 








Distance Stem/Calyx - Center of Gravity 


Figure 5. Normalized Error vs. Distance from C.G. 


Normalized Error vs Angular Location 
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Figure 6. Normalized Error vs. Angular Location 


| The histogram of the errors obtained by the algorithm are shown in Figure 7. The error is 
_ Clearly not normally distributed. Most of the errors in the location estimation are located around 


20. This is approximately the spacing between laser lines in the image. Therefore, it seems 


that there is a clear influence of the resolution of the structured lighting on the performance of 
_ the algorithm. The structured light pattern projected onto the apples in this work was set by the 


. 


laser lenses. Therefore, it was not possible to verify this conclusion. The effect of structured 


_ light resolution on the performance of the algorithm is left for future work. It is important to 


note that the processing time of the algorithm would be highly increased by adding more 
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projections in the image. Therefore, there will be a trade off between accuracy and speed of 
the algorithm. 






Histogram 
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Figure 7. Error Histogram 


*62 non detected images are not shown on the histogram. 


The performance measure was the distance between the actual stem/calyx location and the 
location estimated by the algorithm. Therefore, it is important to examine the error associated 
with the actual stem/calyx location. The first possible error to note is that the calyx and stem 
were considered to be point locations in the image, while in practice they are wider than one 
pixel. The error associated with this assumption is difficult to quantify. A second source of 
error in the determination of the actual location of the stem or calyx is that, it is not possible to 
precisely locate the stem or calyx at the center of each location required. The error associated 
with manual orientation of the apple could range anywhere from 0 to 10 pixels. Therefore, it is 
important to note that there is not improvement for errors of 10 to an error of 15. 


A total of 342 images were collected to test the algorithm. These images were representative 
samples of all possible orientations that the apples can achieve in the prototype transport 
system. There was a total of 62 non-detected apples. From this set,180 images corresponded 
to images with the calyx/stem located within the center part of the apple image (interior circle 
with diameter 40% percent smaller than the original apple). In this region only 14 stems or 
calyxes were not identified. The mean error in the location of the stem and calyx was 20% of 
the radius of the apple, and the standard deviation 12.2% (=17 pixels, s=10 pixels). Most of 
the errors on the outside area were also in this range. However, the number of outliers (large 
errors) and non-detected stems/calyxes increased considerably. 


Conclusion and Future Work: 

Conclusion: 

A setup using structured lighting and a computer vision algorithm was developed to 
automatically identify the stem and the calyx‘on rotating apples. The setup was designed 
considering the environment in which the system would be expected to operate. For this 
purpose, the images used for the analysis were captured using a prototype transport system 
similar to the ones used in apple packing plants. The algorithm described includes the image 
processing steps required to identify the stem and calyx under this setup. 


10 


fi 
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The algorithm seems to be accurate enough with the setup used. However, the accuracy of the 
system is likely to improve by projecting a larger number of lines onto the apples. There were 
approximately nine laser lines projected on the surface of the apple. Therefore, The spacing 
between them was larger, approximately 0.1% of the radius. This resolution seems to be very 
close to the mean value of the error in the algorithm’s estimates. Therefore, it would be 
expected that the accuracy of the system would improve if more lines were projected onto each 
apple. 


Defects on the apple skin are usually present on the cheeks of the fruit. It is expected that 
most of the time there will be a significant separation between the stem or calyx and any 
possible defect. Therefore, an error of 20% of the radius of the apple might be sufficient to 
avoid the misclassification of the stem and calyx. Otherwise, a grid with higher resolution 
should be used. 


Future Work: 

The work presented here did not do any optimization of the physical parameters of the system. 
These parameters are the angle between the structure light source and the focal axis of the 
camera, number of lines projected onto the apple and the resolution of the camera. The former 
parameter has a direct effect on the performance of the algorithm, since it effects the 
sensitivity of the laser projections to changes in surface geometry. The number of lines 
projected onto the apple, and the camera resolution have an effect on the overall performance 
of the algorithm and computation requirements on the algorithm. Therefore, it will be 
necessary to identify the optimum values for these parameters prior to implementing the 
system. Note that the values used for some of the internal parameters of the algorithm will 
need to be changed if the setup is modified. The addition of a hypothesis verification section 
would add more reliability to the algorithm. A second advantage is that the thresholds of some 
parameters could be tightened to ensure that all calyxes and stems are detected. False 
identification could be removed in the hypothesis verification, reducing the number of non- 
detected stems or calyxes. 


This work used Red Delicious apples to evaluate the performance of the algorithm. This was 
done because this variety is the one that most severely tests the assumption of the apples 


_ being spherical. Therefore, it is expected that the system performance will improve for more 


_ possibility of identifying stem and calyx using a pattern classifier. In the introduction and 


spherical varieties. Although, the system parameters might need to be changed for the each 
varieties. 


The technique used to segment most of the defects is a pattern classifier based on reflectance 
at different wavelengths. Therefore, the next focus of the research should be to study the 


literature review sections, it was pointed out that it seems not possible to successfully identify 
all types of defects based using only the surface illuminance levels. However, it might be 
possible to develop a pattern classifier based on other parameters besides gray levels. Some 
other parameters that could be used as input variables for a pattern recognition algorithm could 
be: defect shape, texture, number of blobs, etc. Note that an important consideration to 
Selecting the input parameter for the classifier is that these parameters have to be normalized, 
Since the shape and size of the defects is not constant. 
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BACKGROUND 


Using spectral reflectance to assess fruit quality, both for maturity and for defects, dates 
back to the early 1940’s (Lott, 1943). The earliest work used reflectance spectra to 
measure fruit maturity (Bittner, et.al.1968). Bruised apple pulp consistently had a lower 
reflectance at 600 nanometers (Ingle and Hyde, 1968). It was reported that when water 
replaced air in the intercellular air spaces in plant tissue, the near-infrared reflectance 
decreased (Woolley, 1971). Later, it was recognized that apple bruising ruptured cells 
releasing cell fluid filling the intercellular spaces under the apple skin resulting in a 
reduction of the near-infrared reflectance from the apple surface (Brown et. al, 1974). It 
was further suggested that for processed apples, different wavelengths (wavelenghts is used 
to indicate bands of wavelengths centered about a specific wavelength) could be used to 
discriminate between apple defects, flesh, skin, stem or calyx ends. Further study of the 
mechanics of the trimming and orientation mechanisms was indicated to make sensing 
feasible (Reid, 1976). 


The examination of more than one wavelength provided improvements. A regression model 
found that two selected wavelengths between 350nm - 750nm performed better than a 
single wavelength at classifying bruised and unbruised tissue. Further study found that a 
two wavelength derivative model worked the best (Fuzzen, 1981). Recognizing that not all 
apple flesh browned equally, the optical reflectance of bruised and unbruised apple flesh 
using 16 wavelengths (350nm - 700nm) was measured for five different peeled processing 
varieties (McIntosh, Greening, Spartan, Red Delicious, and Northern Spy). The resulting 
classifier was reduced from 16 to 5 wavelengths and still classified 100% of peeled 


Northern Spy bruised tissue from  unbruised tissue (Pen et. al., 1985). A 
spectrophotometric study of diffuse reflectance between 400nm and 1000nm wavelengths 
compared a single wavelength, two wavelength ratio, two wavelength difference, 
normalized two wavelength difference, and reflectance derivative models to classify 
bruised from unbruised apple tissue (Upchurch et. al., 1990). This study was carried 
further to include early frost damage and red/green areas on the surface of Empires. The 
ratio, normalized difference, and deriative models using more than one wavelength 
classified color and damage better than a single wavelength (Upchurch et. al., 1991). All 
of the above work was performed on labratory spectrophotometers with no attempt to apply 
the techniques to a processing or sorting line. 


Digital images of apples, captured using a video matrix camera and near-infrared (NIR) 
light, were used to compute a linear discriminant classifier capable of detecting bruises on 
Red Delicious apples as accurately as a human inspector (Graf, 1982). An attempt to 
apply a quadratic discriminant classifier to NIR images, captured with a line scan camera, 
failed to handle the variability of the data (Taylor, et. al. 1985). Conversion of Taylor’s 
algorithm from a quadratic to a linear classifier did not improve the performance. A 
nonstatistical threshold-thinness ratio image processing algorithm had a lower grade-by- 
grade classification error than Graf’s algorithm. Apple orientation is important, as part of 
the inspection process (Rehkugler et. al., 1986) (Throop et. al., 1988). Processing of NIR 
images of defects other than bruising segmented some defects successfully (Rehkugler et. 
al., 1989). An attempt to apply the two wavelength models using a camera and two 
bandpass filters instead of a spectrophotometer did not perform the same way as with a 
spectrophotometer . It was observed that reflectance characteristics captured through a 
camera/lens/filter system produced different results than those achieved with the 
spectrophotometer (Upchurch et al, 1990). 


Electronic inspection of apples will be dependent on electronically capturing images, and 
based on reflectance spectral data, quickly surface and subsurface damage by type and as 
difference from undamaged tissue. The reflectance characteristics of a few defects that 
damage apple tisssue have been fully investigated. This study was designed to further 
advance electronic inspection of apples by using a lens/camera system to look at reflectance 
spectral characteristics of many different types of damaged tissue. 


OBJECTIVES 


The objectives of this study were to gather detailed visible and NIR spectral reflectance 
data for a large number of defects found in five varieties of apples and to analyze these 
data. The data were to be captured with imaging system that could be used in an inspection 
station; a matrix camera equipped with lens and optical filters. A classifier was to be 


developed that would determined which wavelengths of light best separate defects and good 
apple tissue based on the computed Mahalanobis distance and the classification error. 


MATERIALS AND METHODS 


During the 1995 harvest season, different apple cultivars with defects were gathered from 
the experimental orchards at the Geneva Experiment Station, Geneva, NY and at the 
Appalachian Fruit Research Station, Kearneysville, WV (Table 1). The apples were held in 
cold storage until 24 hours before image capture. 


Table 1. A list of defects and cultivars from which reflectance spectra was gathered. 


Defect Red Golden Crispin McIntosh Empire 
Delicious Delicious 

Bitter Pit x 

Blister Spot xX 

Bruise x Xx 

Codling Moth x X 

Corking x 

Fly Speck x x 

Hail Damage x 

Powdery Mildew xX x 

Leaf Roller x x 

Puncture(New) x x x 

Puncture(Old) xX 

Rot x 

Russet x x 

Scab x X 

Scald xX 

Sting x x x 

Sunburn x 

Sooty Blotch x x 


Two camera/lens/adjustable filter combinations were used. System 1 consisted of a CCD 
monochrome camera (Model 4900, Cohu Electronics, San Diego, CA) with its gain 
adjusted to full and equipped with a 25mm lens with the aperture set at f1.4. A tunable 
filter system (Model: VS-VIS3-20-MC-20-SQ, Cambridge Research & Instrumentation 
Inc., Cambridge, MA) was placed in front of the camera lens. The filter had variable 
attenuation control and was adjusted for 29 different wavelengths between 460nm to 


750nm, both of which were controlled from the serial port of the image capture computer. 
For each wavelength selected the filter had a 20nm bandpass and a 20nm aperture 
diameter. System 2 consisted of a CCD monochrome camera (Model 4800, Cohu 
Electronics, San Diego, CA) equipped with a 25 mm lens with the aperture set at f1.4. A 
tunable filter system (Model:VS-NIR1-35-MC-20-RN, Cambridge Research & 
Instrumentation Inc., Cambridge, MA) was placed in front of the camera lens. The filter 
had variable attenuation control and was adjusted for 29 different wavelengths between 
750nm to 1030nm, both of which were controlled from the serial port of the image capture 
computer. For each wavelength selected the filter had a 35nm bandpass and a 20nm 
aperture diameter. A white teflon cylinder 76 mm diameter by 76mm high was placed in 
the field of view of both systems. For each wavelength tested, the correct attenuation 
setting for the neutral density portion of the variable filter allowing a camera output of 175 
out 255 grey levels was recorded in a look up table. This assured that the camera output 
was normalized to a standard output grey level while viewing the teflon cylinder regardless 
of changes in camera response due to wavelength or of changes in the transmission 
characteristics of the variable filter. System 1 and system 2 were each controlled by 
different PCI bus Alpha 275 MHz processor equipped computers (Model SYS- 
AXP275AN, BTG Micro Systems, Fairfax, VA and Model Cobra Cab 275, Carrera 
Computers Inc., Laguna Hills, CA) with frame grabber cards (Raptor-PCI, BitFlow, 
Woburn, MA) installed for image capture. Filter control was through the serial port via a 
subroutine running under image capture and processing software (Quantim, ZEDEC 
Technologies Inc., Burlington, NC). For every apple tested, this combination provided 
sequential filter adjustment and image capture for 58 images continuously for each of the 
different wavelengths tested. A total of 10,200 images 256 pixels high by 256 pixels wide 
were captured in this study. Images were stored on a rewriteable 650 MB/disk PD/CD- 
ROM drive (Model:LF-1004AB, Panasonic Communications & Systems Co., Secaucus, 
NJ). 


Diffuse lighting was produced by placing the apples inside of a 222mm diameter 
cylindrical white acrylic diffuser 406mm high with four tungsten lamps placed around the 
outside. The 40 watt lamps (model 40A, 40W, General Electric, Cleveland, Ohio) were 
placed in a 305mm per side square centered about the outside of the diffuser at a height of 
184mm (Figure 1). To minimize 60 Hz fluctuations, the adjustable 120 VAC to the lamps 
was converted to adjustable 160 VDC (4V pk-pk ripple) by using a 10A bridge rectifier 
(GBPC-1004, General Instruments) and a capacitor filter (80 uf 450 VDC, 36DX- 
800F450DC, Sprague). 


The average grey level intensity of undamaged and damaged areas 4 pixels wide by 4 
pixels high were interactively found on the apple images. This average grey level was 
divided by an average grey level of a corresponding spatial area from the teflon cylinder 
image for each wavelength tested with the resulting quotient considered as a measure of 
reflectance. Thus, the reflectance of teflon was considered to have a value of 1. 


Although only 3 - 4 apple images were captured, many different areas of damage were 
present within an image such at 10 to 30 of defects were observed in the 3-4 apples. 
Discriminant analysis of the two classes of tissue (damaged and undamaged) was 
performed (Minitab 9.0, Minitab Inc., State College, PA). The classification error in 
percent and Mahalanobis distance between class means was recorded for each 10nm change 
in wavelength between 460nm and 1030nm. Whenever the classification error was the 
smallest for both classes, the Mahalanobis distance was the greatest indicating the greatest 
reflectance difference between damaged and undamaged tissue. 





Figure 1. Picture of the camera/filter, diffuser, and lamps used to capture images. 
RESULTS AND DISCUSSION 


Bitter Pit 

Bitter pit starts as an internal disorder and later causes external blemishes. Its cause is 
related to the competition between fruits and leaves for water and nutrients. Before harvest 
small areas of cortical tissue at the terminals of branched vascular bundles near the skin 


appear water soaked. Later the spots can take on a deep-red color or bright green color. 
After harvest the affected cells die and lose moisture causing the skin to sink. The skin 
over the area becomes gray, brown, or black. If the skin is peeled, small brown dry 
spongy tissue is found (Pierson et. al., 1971). The reflectance curves show both the color 
change of the skin as a decrease in reflectance, a decrease in reflectance for 700nm - 
830nm due to scattering caused by the damaged cells, and the cellular dehydration for 
damaged with respect to undamaged tissue as an increase in reflectance particularly near 
the water absorption band at 970nm (Figure 2). 
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Figure 2. Reflectance of bitter pit damaged and undamaged Golden Delicious tissue (left) 
and the results of the discriminant analysis comparing the two classes of tissue (right). 


The greatest reductions in reflectance occurred at 540nm (green) and 760nm. 
Correspondingly the discriminate analysis had the lowest classification error at these 
wavelengths and the greatest separation between the two classes of tissue as indicated by 
the greatest Mahalanobis distance. There is a good possibility that this defect would show 
up early in the harvest season before storage as a reduced reflectance at 970nm even before 
it became visible on the skin because of the excess moisture present. 


Blister Spot 
Blister spot is caused by pseudomonas syringae pv. papulans bacteria which is present on 


both the leaves and fruit during the growing season. Primarily damage is found on the 
surface of Crispin apples where the spots on the skin begin to show up in late July or early 
August. The spots originate at the lenticels and turn brown as affected cells die. The 
reflectance curves show this change in color as well as the light scattering affect for NIR 
caused by the dead cells with the resulting reduction in surface reflectance (Figure 3). 
From the discriminant analysis we see that when undamaged tissue reflectance decreased, 
there was a corresponding decrease in reflectance for blister spot damaged tissue as 
indicated by the uniform Mahalanobis distances between classes for wavelengths of 500nm- 
760nm. For wavelengths above 800nm, blister spots gradually faded until they appeared 
the same as undamaged tissue making classification impossible. 
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Figure 3. Reflectance of blister spot damaged and undamaged Crispin tissue (left) and the 
results of the discriminant analysis comparing the two classes of tissue (right). 


Hail 

Apples tend to outgrow hail damage early in the season with a tendency to become slightly 
misshapen. Late in the growing season, internal bruising can occur resulting in slightly 
sunken spots in the skin with bruised tissue underneath which is usually brown in color and 
sometimes spongy in texture as dehydration occurs. This condition can be confused with 
cork spot, but usually in cork spot the brown spongy tissue is deeper into the flesh. No 
difference in reflectance between hail damaged and undamaged tissue was found for the 
visible wavelengths. From 690nm - 720nm the greatest reduction in reflectance occurs as 
scattering increases due to the damaged cells. As the wavelength increases past 900nm, 
reflectance from some hail damaged areas equals or exceeds that of undamaged tissue due 
to partial dehydration (Figure 4). The discriminant analysis had large classification errors 
for the visible wavelengths resulting from small reflectance differences compared to the 
typical reflectance variations for all tissue classes. Large classification errors for the NIR 
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Figure 4. Reflectance of hail damaged and undamaged Red Delicious tissue (left) and the 
results of the discriminant analysis comparing the two classes of tissue (right). 


wavelengths above 760nm occurred because of the wide variation in depth of damage in 
combination with variable amounts of dehydration. Dehydration is dependent on the length 


of time after damage and the depth of damaged cells. The Mahalanobis distance was the 
greatest for wavelengths 690nm - 740nm with the lowest classification error indicating the 
largest separation between damaged and undamaged classes (Figure 4). 


Cork Spot 
The apples may develop one or more spots slightly sunken with a surface that is never 


brown or black. The corky areas are usually larger than bitter pit and may lie just below 
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Figure 5. Reflectance of cork damaged and undamaged Red Delicious tissue (left) and the 
results of the discriminant analysis comparing the two classes of tissue (right). 


the skin or halfway to the core. The underlying tissue is brown and spongy showing loss of 
moisture. Although there is uncertainity to the cause of this damage, an imbalance of boron 
and calcium may disturb the permeability of the cell walls resulting in the death of certain 
cells influencing development of cork spot. Reflectance of cork spot on Red Delicious 
apples decreases in reflectance at 630nm - 820nm. At 860nm the reflectance of cork tissue 
equals that of undamaged tissue. Above 920nm cork damage has a reflectance greater than 
undamaged tissue due to the loss of moisture within the cells (Figure 5). Cork spot damage 
is very similar to hail damage with the exception that the depth of injured cells tends to be 
consistantly greater in depth. This is verified in the discriminant analysis where the 
classification error is lower for most wavelengths. Similar to hail damage, the Mahalanobis 
distance is greatest for the 690nm - 740nm wavelengths but larger in magnitude (Figure 5). 
The camera system used in this study is not sensitive enough to measure cork damage 
located deeper in the fruit. 


Elyspeck 
Flyspeck are superficial specks of fungal growths occurring in groups of 8 to 50 specks or 


more. The causal fungus, microthyriella rubi which is active in cool humid weather 
develops on mature fruits. The specks can be scrapped off easily although cleaning the total 
apple surface with automatic equipment successfully can be difficult. The reflectance of fly 
speck damage decreased for all NIR wavelengths. For both Red and Golden Delicious (not 
shown), the greatest Mahalanobis distance occurs at 740nm, corresponding with a 


classification error of 0 percent for both damaged and undamaged tissue (Figure 6). Golden 
Delicious exihibited a second Mahalanobis distance peak for wavelengths between 500nm- 
520nm with a classification error of 0 percent. The classification error above 800nm was 
higher for Golden Delicious compared to Red Delicious, indicating that the selection of 
images of reflectance of 740nm would be best for classification of flyspeck damage on both 
Red and Golden Delicious. 
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Figure 6. Reflectance of flyspeck damaged and undamaged Red Delicious tissue (left) and 
the results of the discriminant analysis comparing the two classes of tissue (right). 


Brown Rot 

Brown rot caused by the fungus Monilinia fructicola enters the fruit through either the 
stem/calyx ends, wounds, cracks, or insect-feeding injuries. It produces circular medium 
brown spots into the apple flesh that are soft but not mushy. This fungus can be controlled 
by low temperatures in refrigerated storage unless it has become well established. Once 
established, large areas can become affected over time. The reflectance is reduced for rot 
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Figure 7. Reflectance of brown rot damaged and undamaged Red Delicious tissue (left) and 
the results of the discriminant analysis comparing the two classes of tissue (right). 


damaged tissue for all of the visible and near-infrared wavelengths tested with the greatest 
reduction occurring between wavelengths 700nm - 800nm. This corresponds with the 
greatest distance between between the two classes, damaged and undamaged tissue, as 
indicated by the largest mahalanobis distances and lowest classification errors for this range 
of wavelengths (Figure 7). As the wavelength increased beyond 930nm, some rot damaged 
tissue looked identical to undamaged tissue while in other cases the damage was severe 
enough to cause release of cellular fliud further decreasing reflectance. 


Scab 

Infections of the fungus venturia inaequalis is spread to new fruit and leaves during rainy 
periods in the spring. Cool moist weather conditions can further infect fruit any time in the 
growing season. Irregularly circular scab lesions first appear and become larger turning an 
olive green or brown in color. Highly infected apples can become misshapen. Scab injury 
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Figure 8. Reflectance of scab damaged and undamaged Red Delicious tissue (left) and the 
results of the discriminant analysis comparing the two classes of tissue (right). 


reduced reflectance from 600nm to 930nm. The greatest class separation occurred between 
700nm-750nm as indicated by the greatest values of Mahalanobis distance and with 
classification errors of 0 percent (Figure 8). Some of the scab areas on the Red Delicious 
disappeared above 940nm and could not be discerned from undamaged tissue. Reflectance 
of more severe scab damage on McIntosh (not shown) had a much higher Mahalanobis 
distance value of 65 but still peaking at 740nm. Unlike Red Delicious, all increasing 
wavelengths tested for McIntosh had a 0 percent classification error. 


Sooty Blotch 
Late-summer rains combined with cooler temperatures are conducive for the fungus, 


gloeodes pomigena, to produce dark granular irregular spots with indefinite outlines 
varying greatly in size on the skin surface. The blotches are thin fungal crusts which can be 
removed with scraping or rubbing. In severe cases moisture loss can occur and the spots 
can become sunken in storage. Reflectance of sooty blotch damage general lowered the 
reflectance for all wavelengths tested on Golden Delicious apples. However, the largest 


10 





Mahalanobis distances between classes occurred for NIR wavelengths of 700nm to880nm. 
A minimum classification error of 7 percent occurred for these same wavelengths which 
increased to nearly 30 percent for wavelengths between 920nm and 970nm. The greater 
classification error for 920nm-970nm may be caused by some spots where moisture loss 
has occurred increasing the reflectance for these areas back to that of undamaged tissue. 
The presence overall of a general classification error of 7 to 10 percent can be due to the 
variable thickness and discoloration of the surface spots (Figure 9). Simular results of 
reflectance for sooty blotch damaged Empire apples (not shown) also had the largest 
Mahalanobis distances between classes for 700nm-880nm wavelengths but with O percent 
classification error. 
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Figure 9. Reflectance of sooty blotch damaged and undamaged Golden Delicious tissue 
(left) and the results of the discriminant analysis comparing the two classes of tissue 
(right). 


Scald 

Scald is a physiological disorder where apples develop superficial browning and death of 
the skin during and after removal from cold storage. For some varieties the underlying 
tissue can also be affected. The severity of damage varies from year to year. Reflectance 
from scald damaged regions on Red Delicious was typically less than undamaged tissue for 
all wavelengths tested above 600nm. The greatest difference between classes occurrs 
beyond 880nm at which time the classification error becomes 0 percent. There is a 
corresponding large increase in the Mahalanobis distance. The wavelengths for this large 
class difference tends to coincide with the maximum water absorption band at a wavelength 
of 960nm indicating that the tissue break down may release cellular fluids under the skin 


(Figure 10). 
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Figure 10. Reflectance of scald damaged and undamaged Red Delicious tissue (left) and 
the results of the discriminant analysis comparing the two classes of tissue (right). 


Bruise 

Skin covering bruised areas is normal or slightly discolored but under this skin are raptured 
cells where the surrounding tissue turns a light brown and is filled with the fluid from the 
broken cells. If placed in storage, this fluid eventually evaporates through the skin causing 
the bruised tissue to become dry and spongy. This study looked at fresh bruises that were 
24 hours old. The reflectance of bruised tissue was lower than undamaged tissue for all 
wavelengths tested above 680nm. The classification error was 5 percent or less for all 
wavelengths above 800nm for Red Delicious (shown) and 850nm for Golden Delicious (not 
shown). The lowest classification error and greatest Mahalanobis distances occurred at 
970nm, similar to scald. This wavelengths again coincide with the maximum water 
absorption band at 970nm indicating light absorption by the released cellular fluids under 
the skin (Figure 11). 
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Figure 11. Reflectance of bruise damaged and undamaged Red Delicious tissue (left) and 
the results of the discriminant analysis comparing the two classes of tissue (right). 


Leaf Roller and Codling Moth 


se 


The larvae of the red-banded leaf roller consume areas of skin and outer flesh. This results 
in areas of dark colored dead cells which usually have dryed out. Although codling moth 
larvae can bore directly into the apple core causing severe damage, usually they eat some 
skin and outer flesh before they consume a lethal dose of poison and die. The codling moth 
spots are smaller but more numerous than leaf roller injury. They both have similar 
reflectance properties and are reported together. The reflectance of leaf roller damage on 
Golden Delicious (shown) look identical to the codling moth injury on Crispin (not 
shown). The reflectance is reduced over the visible regions of green to red as well as 
wavelengths between 700nm - 880nm. The reflectance than increases to that of undamaged 
tissue and in cases where drying has occurred exceeds undamaged tissue reflectance above 
the water absorption wavelength of 970nm. Reflectance from damage on Red Delicous (not 
shown) does not have a usable decrease in reflectance in the visible wavelengths but has 
very similar reflectance characteristics for NIR. The discriminant analysis showed 
corresponding large values of Mahalanobis distances and 0 percent classification error for 
the wavelengths where the reflectance for damaged areas was the lowest (Figure 12). 
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Figure 12. Reflectance of leaf roller damaged and undamaged Golden Delicious tissue 

(left) and the results of the discriminant analysis comparing the two classes of tissue 


(right). 


Russet 

Russeting is a skin damage where areas become light tan in color with a slightly rough 
texture. It is caused by virus infection, insect stings, early frost damage at time of blossom, 
and by the fungus commonly refered to as powdery mildew. The damage is usually only 
skin deep affecting only the appearance and does not affect the eating quality of the fruit. 
The reflectance from russet damaged Red Delicious tissue increases for wavelengths 
around 540nm and decrease for wavelengths between 680nm to 740nm. However, the 
differences are small compared to the apple-to-apple variability resulting in very small 


ANS 


Mahalanobis distances with a large classification error for all wavelengths (Figure 13).The 
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Figure 13. Reflectance of russet damaged and undamaged Red Delicious tissue (left) and 
the results of the discriminant analysis comparing the two classes of tissue (right). 


reflectance from russet damaged Golden Delicious tissue decreases for wavelengths around 
540nm and for wavelengths between 680nm to 740nm. Again, the differences are small 
compared to the apple-to-apple variability resulting in very small Mahalanobis distances 
with a large classification error for all wavelengths (Figure 14). 
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Figure 14. Reflectance of russet damaged and undamaged Golden Delicious tissue (left) 
and the results of the discriminant analysis comparing the two classes of tissue (right). 


If the reflectance for a wavelength of 540nm is used to segment russet, it will require a 
different algorithm for green/yellow apples versus solid red apples. At 540nm it will be 
difficult to segment russet damage on green/red cultivars such as Empires or McIntosh 
because russet damage on solid red apples has a greater reflectance than undamaged tissue 
compared to solid green/yellow apples where russet damage reflectance is less than 
undamaged tissue. 
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Sunburn 

Sunburn is a gold or bronze coloring of the skin as a result to exposure to long periods of 
intense sunlight. None of the cells in the skin are killed and the flesh underneath, if 
anything, becomes slightly firmer. Apples with this type of injury can rapidly soften in 
affected areas upon removal from cold storage. Little variation in reflectance is apparent 
with only minor color variations for the green through red wavelengths. Based on the 
Mahalanobis distance and classification error, only some areas of sunburn where the red 
color of the Red Delicious is altered holds any hope for segmentation and detection (Figure 
15): 
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Figure 15. Reflectance of sunburn damaged and undamaged Red Delicious tissue (left) and 
the results of the discriminant analysis comparing the two classes of tissue (right). 


Sting 

The reflectance of damage from stinging insects on apples, in this case Red Delicious, is 
highly dependent on the length of time before harvest that the damage occurred. If early in 
the growing season, the area takes on a russet like appearance with healing underneath. 
Later in the season, the affected cells die and dehydrate much like coddling moth or leaf- 
roller damage. The reflectance for all types of sting damage is reduced between 700nm- 
790nm. The reflectance beyond 860nm for early russet like damage appears equal to 
undamaged tissue. Late season damage dehydrates and increases in reflectance beyond 
920nm (Figure 16). For wavelengths of 740nm-760nm, classification error was O percent 
and the largest Mahalanobis distance occurred regardless of early or late season damage. 


Puncture (new) 


Punctures, particularly those caused by stems, are a common form of damage. The 
breakage of the skin allows bacteria and fungus to enter the fleshy portion of the apple 
causing rapid decay. Reflectance for all NIR wavelengths decreased for punctured areas 
caused by stems (Figure 17). A large Mahalanobis distance between classes with a 
corresponding 0 percent classification error indicates that all that is required for puncture 
segmentation in NIR is the correct pixel resolution for the small damage size. 
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Figure 16. Reflectance of insect sting damaged and undamaged Red Delicious tissue (left) 
and the results of the discriminant analysis comparing the two classes of tissue (right). 
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Figure 17. Reflectance of newly puncture damaged and undamaged Red Delicious tissue 
(left) and the results of the discriminant analysis comparing the two classes of tissue 


(right). 


CONCLUSIONS 


For the eighteen defects reported, reflectance images at 4 wavelengths give the greatest 
contrast between damaged and undamaged tissue. Russet, sunburn, early sting, and blister 
spot damage showed the greatest Mahalanobis distances for 540nm wavelengths and a low 
classification error. These types of damage are typically limited to the skin and usually 
alter color only. The change in reflectance for sunburn and russet where very small and 
will be difficult to differentiate from undamaged tissue regardless of the wavelength used. 
Bitter pit, blister spot, hail, cork, fly speck, brown rot, scab, sooty blotch, leaf roller and 
codling moth damage showed the greatest contrast between damaged and undamaged tissue 
for reflectance images of 750nm. The defects which showed reflectance responses of 
greater than undamaged tissue such as cork, bitter pit, leaf roller, stored bruises (not 
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shown), codling moth, and some early season insect stings occurred for 1030nm 
reflectance images. This wavelength is an absorption band for water thus if different 
moisture conditions appear in the damaged area compared to moisture conditions for 
undamaged tissue, it would be a useful wavelength for defect segmentation. Finally, scald 
and new bruising had greatly reduced reflectance for damaged compared to undamaged 
areas at 970nm. This is due to a combination of scattering due to the ruptured cells in 
combination with the absorption due to the fluid released from the ruptured cells. 


Specific narrow and broad bands of wavelengths of light have been defined by discriminant 
analysis where the greatest contrast between classes of tissue in apple images can be 
expected for a number of different types of damage and different cultivars. However, this 
is only part of the information required for developing an electronic sorting system capable 
of detecting many kinds of damage and identifying each one from each other. This process 
of identification will require inclusion of pattern recognition as well as reflectance 
information and an algorithm approach allowing training to handle the wide range of defect 
types all on many different cultivars. The data presented in this report represents only the 
first part of the process. 
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Abstract 


Surface blemishes of various apple varieties were analyzed by their reflectance characteristics between 460 and 1130 
nm. Normalized reflectance data were collected at 10 nm increments with liquid crystal tunable filters. Data were 
utilized as input values for various pattern recognition models specifically multi-layer back propagation, unimodal 
Gaussian, K-nearest neighbor and nearest cluster algorithms. Partitioning data into 50:50 training and test sets, 
correct classification in separating unflawed versus blemished areas ranged from 62 to 96% (Year I) and from 73 to 
85% (Year II). The algorithm which yielded the highest correct classification was the multi-layer back propagation but 
minor variation was found for number of hidden nodes or neural net architecture. © 1998 Elsevier Science B.V. All 


rights reserved. 


Keywords: Artificial intelligence; Grading; Neural network 





1. Introduction 


For fresh fruits and vegetables, surfaces blem- 
ishes are considered grade-lowering characteristics 
under USDA grade standards. Such defects may 
be the result of environmental conditions, insects, 
pathogens or physiological disorders. In some 


* Corresponding author. Tel.: +1 941 9561151; fax: +1 
941 9564631. 


cases such defects actually may degrade internal 
quality while, in other mstances, the blemished 
portion is only a cosmetic flaw. For eastern 
United States apples, four grades are defined 
ranging from US Extra Fancy to US Utility 
(USDA, 1976). Some defects, such as russet and 
sunburn, are observed readily in manual grading, 
while others, e.g. bruising, are not easily dis- 


cerned. To eliminate the arduous task of grading 
and provide a more uniform product, automated 
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Table | 

Application MLBP Bayes KNN 
Target recognition" 90.7 Ses) 9O7 
Map feature? data 91.2 89.3 91.9 
Handwriting* 94.8 — 94.8 
Carrot defect® 89.0 81.3 — 
Potato defect* 67.8 — — 





ae — 71.6 = 


Bayes, unimodal Gaussian; MLBP(H), Multilayer back propagation (hardware); KNN, k nearest. 
* Ruck et al. (1989): ° Daud et al. (1995); © Lee (1991); “ Howarth and Searcy (1991); © Deck et al. (1995). 


grading via solid-state camera sensing has been 
advanced (Rehkugler and Throop, 1986; 
Upchurch et al., 1994; Tao et al., 1995: Guedalia, 
1997). Also, other sensors such as X-ray (Schatzki 
et al., 1997; Shahin and Tollner, 1997) and infra- 
red sensing (Moons et al., 1997) have been pro- 
posed for detection of surface flaws and internal 
quality. 

In conjunction with sensor technologies, clas- 
sification strategies have been advanced. An early 
treatise on the subject was presented by Duda and 
Hart (1973). Their work centered on Bayesian 
decision theory and non-parametric techniques 
such as Parzen windows. More recently, tech- 
niques such as neural networks (Rogers and 
Kabrinsky, 1991; Schalkoff, 1992; Lawrence, 
1994) have been proposed and applied to fresh 
fruit and vegetable machine-based grading 
(Miller, 1992: Deck et al., 1995; Guedalia, 1997; 
Heinemann et al., 1997; Miller and Drouillard, 
1997). The neural network approaches have been 
cited for the advantages of knowledge plasticity to 
changing inputs and outputs, fault tolerance, 
noise immunity and interpolation/extrapolation 
capabilities. Comparison of statistical classifiers 
and neural networks has been undertaken for 
numerous application as compiled in Table 1. As 
exhibited in this table, such pattern recognition 
comparison has been inconclusive in identifying a 
single technique that exhibits robust superiority 
with respect to correct classification of multi-fea- 
ture data sets. 

With no clear superiority among classification 
techniques, software products have been devel- 
oped to implement various pattern recognition 
approaches for comparison. Based on reflectance 


levels at various wavelengths, various pattern 
recognition models were tested for their capability 


to segregate surface flaws of apples. Specific ob- - | 


jectives were: 

1. To develop baseline reflectance data, 460- 

1130 nm, for typical surface defects found on 

apples grown in the eastern United States for 

fresh market consumption; 

To compare pattern recognition techniques 

that could be implemented for categorizing 

specific defects and segregating sound versus 

defective apples; and 

3. To establish optimal wavelength(s) for specific 
and general defect determination. 


to 


2. Materials and methods 


During the Fall harvest seasons of 1995 and 
1996, various apple varieties were collected from 
trees at the USDA Appalachian Fruit Research 
Station, the WVU Kearneysville experimental 
farm, and from commercial packers. Varieties in- 
cluded in one or both years were: ‘Empire, 
‘Gala’, ‘Golden Delicious’, “Granny Smith’, ‘Red 
Delicious’, ‘Rome’, ‘Red Stayman’ and ‘York. 
The defects encountered were: bitter pit, blister, 
bruise, cork, fly spec, hail damage, insect damage, 
punctures, rots (reds), russet (reds), scab, scald 
and sooty blotch. For a given defect, five to ten 
different areas were scanned as well as an equal 
number of corresponding unflawed areas. Stem 
and calyx areas were scanned also since their 
geometry and resultant shadowing can be inter- 
preted as defects in machine vision analysis 
(Crowe and Delwiche, 1996). 
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The two camera/lens and adjustable filter ar- 
rangement used to acquire the spectral data has 
been described by Aneshansley et al. (1997). Two 
electronic tunable filter systems were used to span 
the visible, 460-750 nm, and near infra-red spec- 
tra, 750-1030 nm. The bandwidth of the visible 
filter was 20 nm while the near infra-red band- 
width was 35 nm. Utilizing 10 nm steps, 58 selec- 
tively filtered images were acquired for each 
scanned area. All reflectance data were first nor- 
malized based on a standard reflectance from 
either a Teflon sphere or cylinder which exhibited 
minimal absorption bands over the selected spec- 
tra. The 58 normalized reflectance values for a 
given defect or unblemished area constituted an 
input data set. All data sets were then normalized 
by: Value,o¢m = (Value — Value,,.,)/(Walueiax — 
Value,,..)- 

The pattern classification was conducted on the 
normalized data using two software packages 
(PRW Vers. 2.1; Unica Technologies, Brighton, 
MA, and NeuroShell2; Ward Systems Group, 
Frederick, MD). Data were split into training and 
test sets of equal size. The PRW software offered 
the versatility of comparing models other than 
multi-layer back propagation (MLBP) methods 
and included the unimodal Gaussian (UG), K- 
nearest neighbor (K NN). and nearest cluster (NC) 
techniques. For the NeuroShell2 software, testing 
was extended on classification success to more 
complex neural network configurations of multi- 
ple hidden slabs and a three-layer jump model. 

The data were analyzed separately for each 
season with all defects included, 1.e. two category 
results of either sound or defect classification. 
Nine-hundred and eighty-two, and 1316 scanned 
spectra were available for the first and second 
seasons, respectively. Further analysis of the data 
was performed to classify each defect by its dis- 
crete reflectance signature. Also, contribution fac- 
tors were extracted to compare with a separate 
assessment via discriminant analysis (Aneshansley 
et al., 1997). For pattern recognition technique 
comparison of the overall data, the following test 
conditions were selected: multi-layer back propa- 
gation (MLBP @ 10, 20, 30, 40 and 50 hidden 
nodes), nearest cluster (NC @ 1, 5, 10, 25 and 50 
clusters), and K-nearest neighbor (KNN @ 1, 2, 


5, 10 and 25 neighbors). For comparison to more 
traditional Bayesian modeling, an UG was in- 
cluded. General protocol in selection of numbered 
clusters, nearest neighbors. etc., and their interac- 
tions with other model variables, is detailed by 
Kennedy et al., 1995. Typical epoch process time 
on a 75-MHZ Pentium™ computer was approxi- 
mately 1.1 s for the 50 hidden node model cou- 
pled to 58 inputs, two outputs for the MLBP. The 
contribution factors or weights, were extracted for 
each of the MLBP models. A cross correlation 
was performed among the contribution factors for 
each season’s MLBP models. 


3. Results and discussion 


The train and test approach 1s commonly im- 
plemented in classification assessment. Some re- 
searchers have suggested a third evaluation 
through a validation set to overcome bias and 
over-learning (Kennedy et al., 1995). However, as 
this study was directed at model comparisons, it 
was decided that a test/train breakdown was ade- 
quate. All data were normalized twice; first in 
instrumentation calibration and then in model 
preprocessing. 

In an overall approach, all defects, including 
bruising, were categorized together. Therefore, the 
neural network outputs were binary with 0 for 
sound or | as defect. Fifty-eight inputs, represent- 
ing the reflectance data, were used for both the 
1995 and 1996 apple data. Random number gen- 
eration was used to develop the test and train 
data sets. Default transfer functions for MLBP 
were hyperbolic for PRW and logistic for Neu- 
roShell2. Tables 1 and 2 present the data for the 
two seasons. Both training and test data were 
tabulated with respect to percent correct classifi- 
cation. Correctly classified objects were denoted 
as either acceptable as acceptable (A/A) or reject 
as reject (R/R). Error types, i.e. (A/R) acceptable 
as reject (true state) or (R/A) reject as acceptable 
(true state), were extracted. It would be antici- 
pated that a singular NC model would degenerate 
to 50% and that the KNN (1) model would yield 
100% in training. The first constitutes an equal 
probability selection, the latter self identification. 
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Table 4 
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Percent correct classification levels when classified for specific defect 








eee 





Model 1995 test 

Training West 

A/ R/ ER A/A R/R 
MLBP (10) 98.8 05.9 70.9 95.4 94.1 
NC (10) On 60.5 AS IN) 68.0 
KNN (10) 98.8 95.8 70.9 98.3 81.3 
UG 91.2 92.4 69.2 Bill pl 


1996 test 

Training Test 
CR A/A R/R CR A/A R/R CR 
62.1 98.3 78,2 41.6 96.8 We 29.6 
21.9 98.3 64.6 ead) 96.4 63.4 11.6 
42.0 97.0 70.0 30.5 96.8 69.4 25.0 
68.3 98.3 lee 42.0 98.0 76.5 Sone 





A/A, Acceptable as acceptable; CR, Correct category within reject; R/R, Reject as reject. 


Since the singular NC model provides no training 
potential, it was not included in statistical com- 
parisons that follow. 


3.1. 1995 apple data 


Means from the test averages were compared 
through a Duncan’s multiple range analysis tests 
with significant difference among test means indi- 
cated in Table 2. The NC models with five and 
ten clusters did not perform as well as the other 
pattern recognition models. With these NC mod- 
els of lower performance, it was not clear why the 
error rates were higher since the type of errors, 
R/A and A/R, were reversed between these two 
NC model results. The degradation from training 
to test results; moted as % change of the last 
column, was small with values greater than 4% for 
only two of the KNN models. 

Input data were also presented with each defect 
representing a separate class. Each type of model 
was tested in this approach (Table 4). Percent of 
fruit in the correct reject class (CR) was identified. 
The highest CR value was 68.3% for the UG 
model. However, other than the NC (10) model, a 
high percent of the fruit was still segregated cor- 
rectly with respect to a two-category criteria of 
accept/reject. 


3.2. 1996 apple data 
Means from the test averages were compared 


again through a Duncan's multiple range analysis 
and differences between means indicated superior 


performance of all MLBP models and KNN mod- 


els of one, five and ten nearest neighbors (Table 
3). The degradation levels between training and 
test results were higher than in the 1995 analysis. 
These “% change degradation figures provide some 
indication of model robustness. Percent change in 
KNN of one and two nearest neighbor was 
greater than 10% while the MLBP, NC and UG 
models had the smallest degradation ranging from 
1.1 to 5.5%. In general, all models showed poorer 
performance in the second season and may be 
attributed to a larger data set with more diverse 
defects included. The CR analysis (Table 4) indi- 
cated a drop in correct identification of specific 
defects. CR levels were less than 60% in all cases 
and were approximately 50% of those obtained in 
the 1995 test model results. Overall correct clas- 
sification remained high, as with the 1995 data, 
with A/A values ranging from 96.4 to 98.0% and 
R/R values were from 63.4 to 76.5%. 


3.3. Neural network comparisons 


An area of current research interest is the 
configuration, i.e. structural interconnection, of 
the neural network for back propagation. Archi- 
tectures with advanced feedback or feedforward 
elements include the Jordan Elman and jump 
connection models as well as multiple hidden 
layers with varied transfer functions. In this re- 
search, the Jump step approach and the MLBP 
with different transfer functions in the hidden 
layer were assessed. In the latter, three hidden 
slabs were structured with either 7 or 17 nodes per 
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Fig. 1. General configuration for MLBP with three transfer 
function in hidden node layer. 


slab. Transfer functions were hyperbolic tangent, 
Gaussian and Gaussian complement (Fig. 1). In 
the jump step configuration, the inputs were 
passed both to the hidden layer and to the output 
layer (Fig. 2). The comparative results based on 
the 1995 and 1996 apple data are presented in 
Table 5. No superiority was found for the more 
complex models evaluated. A statistically signifi- 
cant difference, at the 5% level, was found be- 
tween years and the standard MLBP neural net 
models. The latter occurred even though the 
learning rate and momentum variables were iden- 
tical. Overall correct classification in testing 
ranged from 92.3 to 96.2'% (1995 data) and trom 


Acceptable 





50 hidden Output 
nodes, logistic nodes, logistic 


58 input 
nodes, linear 


Fig. 2. General configuration for MLBP set for a jump step 
connection. 


Table 5 
Test comparison results for various MLBP models 





Technique Variable OS oIG 
MLBP (PRW) 10 OCS SLO 
20 OS mor, 

30 94.2 83.8 

40 94.5 83.9 

50 OOo. 

MLBP (WARD) 10 9310 eooeS 
20 SOMO sey, 

30 02-6. 83.0 

40 Dato Lai) 

50 O23 83:6 

MLBP?® (variable transfer func- 3(@7 93.2 89.4 

tion) 

3@17 940 88.4 

MLBP* (jump) 50 88.8 





“Ward Systems software. 


82.5 to 89.4% (1996 data) among the MLBP 
standard and advanced models. 


3.4. Contribution factors 


An important aspect of this study was to ascer- 
tain if certain wavelengths could be selected for 
bruise and blemish defect of apple. An obvious 
value to extract for that analysis would be contri- 
bution weights associated with the input values. 
These factors represent a normalized contribution 
with respect to the input variables. In this study, 
the inputs were the 58 reflectance values taken at 
10 nm increments. These factors using minimum, 
maximum, and smoothed average results for the 
seven neural net models versus scanned spectra 
were plotted in Figs. 3 and 4 for the apple data of 
1995 and 1996. Peaks occurred at 740, 600 and 
530 for the first season, and 710 and 460 in the 
second season. The bandpass from 690 to 750 
provided the largest contribution factors. This 
spectral segment was in the overlap between visi- 
ble and near infra-red, and indicated the impor- 
tance of scanning beyond the visible spectrum. A 
cross correlation between the models indicated the 
contribution factors was consistent between the 
models for each year (Table 6). Values for r 
ranged from 0.78 to 0.96. An overall cross corre- 
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Fig. 3. Contribution factors for each input value for various MLBP models, 1995 data. 


lation for both years was statistically significant 
but not as high, r= 0.59. 


3.5. Comparative analysis 


Utilizing another data analysis procedure, dis- 
criminant analysis (Aneshansley et al., 1997), a 
comparison was made for specific defect results of 
the neural network models. For an individual 
defect, contribution factors were determined and 
compared the Mahalanobis distance of the dis- 
criminant approach. Again, a smoothing function 
was applied to the contribution factors versus 
wavelength data. The apple scald defect was se- 
lected for analysis as it’s peak Mahalanobis dis- 
tance was greater than 900 nm, beyond the 
principal peak at 690-750 nm found for most 
apple defects (Figs. 3 and 4)..The correlation 
value between a smoothed contribution factor and 
the Mahalanobis distance associated with each 
wavelength was 0.92. This high correlation indi- 
cated the need for either data smoothing or boot- 
strap implementation as_ individual neural 
network analyses, at 10-50 hidden nodes, had 


much lower values ranging from 0.16 to 0.45. The 
high correlation coefficient provided confidence 
that the neural networks could function on the 
more limited data sets of an individual defect. 


4. Conclusion 


The optical analysis system, configured by using 
two CCD black and white cameras with electri- 
cally tunable filters to scan visible and near infra- 
red, provided discrete reflectance curves to 
analyze defects found in eastern United States 
apples. It was necessary to normalize the reflec- 
tance values of the apples based on a Teflon 
standard. Utilizing both sound and defect areas 
from various grade lowering defects, a database 
of reflectance values was developed at 10 nm 
increments from 460 to 1130 nm. These reflec- 
tance values were used in pattern recognition 
models based on MLBP, NC, KNN and UG 
models. With a more limited data set, all models 
exhibited higher success rates in classification of 
the 1995 season’s data. Overall, the MLBP pro- 
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Fig. 4. Contribution factors for each input value for various MLBP models, 1996 data. 


vided the highest correct classification rates, rang- neural network models of jump connections and 
ing from 83 to 85% for the 1996 data and from 94 multiple transfer functions in the hidden layer did 
to 96% for the 1995 data. The more complex not provide higher success rates. Contribution 
Table 6 


Cross correlation value (r) for contribution factors based on seven MLBP models, 460-1030 wavelength 








Model MLBP (10) MLBP (20) MLBP (30) MLBP (40) MLBP (50) MLBP (3@7) MLBP (3@17) 





Apple95 data 
MLBP (10) 1.00 


MLBP (20) 0.89 1.00 

MLBP (30) 0.82 0.83 1.00 

MLBP (40) 0.89 0.91 0.81 1.00 

MLBP (50) 0.86 0.89 0.78 0.91 1.00 

MLBP (3@7) 0.81 0.79 0.83 0.87 0.83 1.00 

MLBP (3@17) 0.84 0.87 0.81 0.88 0.85 0.84 1.00 


Apple96 data 
MLBP (10) 1.00 





MLBP (20) 0.95 1.00 

MLBP (30) 0.92 0.94 1.00 

MLBP (40) 0.95 0.96 0.93 1.00 

MLBP (50) 0.92 0.94 0.90 0.96 1.00 

MLBP (3@7) 0.88 0.91 0.91 0.92 0.90 1.00 

MLBP (3@17) 0.91 0.93 0.91 0.94 0.93 0.89 1.00 





Overall cross correlation results of the averages of Apple95 and Apple96 data: 0.59. 
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factors, established from the MLBP models, indi- 
cated the most significant wavelengths were in the 
690-750 nm range and 530 nm when considering 
data from both seasons. 


Acknowledgements 


W.M. Miller would like to recognize the finan- 
cial support from the University of Florida’s Fac- 
ulty Development Program and the USDA-ARS 
to undertake this study. William Anger, Engineer- 
ing Technician, USDA-AFRS, contributed signifi- 
cantly to the instrumentation and data acquisition 
portions of the program. Greg Drouillard, Sr. 
Engr. Tech., University of Florida is acknowl- 
edged for his computer work in data reduction 
and analysis. Florida Agricultural Experiment 
Station Journal Series No. A-00384. 


References 


Aneshansley, D.J., Throop, J.A.. Upchurch, B.L., 1997. Reflec- 
tance spectra of surface defects for apples. In: M. Salius 
(Ed.), Sensors for Nondestructive Testing, measuring the 
Quality of Fresh Fruits and Vegetables. NRAES-97. 
NRAES, Ithaca, NY, pp. 143-160. 

Crowe, T.G., Delwiche, M.J., 1996. Real-time defect detection 
in fruit-part II: an algorithm and performance of a proto- 
type system. Trans. ASAE 39 (6), 2309-2317. 

Daud, T., Duong, T., Tran, M., Langenbacher, H., Thakoor, 
A., 1995. High resolution synaptic weights and hardware-in- 
the-loop learning. Vol. 2424. SPIE— International Society 
for Optical Engineering; Bellingham, WA. pp. 489-500. 

Deck, S.H., Morrow. C.T., Heinemann, P.H., Sommer, H.J., 
1995. Comparison of a neural network and traditional 
classifier for machine vision inspection of potatoes. Appl. 
Engr. Agric. 11 (2), 319-326. 

Duda, R.O., Hart, P.E., 1973. Pattern Classification and Scene 
Analysis. Wiley, New York. 

Guedalia, 1.A.. 1997. Visual based quality sorting of fruit. In: 
M. Salius (Ed.), Sensors for Nondestructive Testing, Mea- 
suring the Quality of Fresh Fruits and Vegetables. NRAES- 
97. NRAES, Ithaca, NY, pp. 232-238. 

Heinemann, P.H., Sommer, H.J., Morrow, C.T., Marrazzo, 
W.N., Crassweller, R.M., He, B., 1997. Machine vision 
based station for grading of “Golden Delicious” apples. In: 
Sensors for Nondestructive Testing, Measuring the Quality 
of Fresh Fruits and Vegetables. NRAES-97. NRAES, 
Ithaca, NY, pp. 239-248. 


Howarth, M.S., Searcy, S.W., 1991. Comparison of Bayesian 
and Neural Network Classifiers for Grading Carrots. ASAE 
Publ. 7012-91. ASAE, St. Joseph, MI. 

Kennedy, R., Lee, Y., Reed, C., Roy, B.V., 1995. Solving 
Pattern Recognition Problems. Unica Technologies, 
Brighton, MA. 

Lawrence, J., 1994. Introduction to Neural Networks Design, 
Theory and Applications. California Scientific Software, 
Nevada City, CA. 

Lee, Y., 1991. Handwritten digit recognition using K nearest- 
neighbor, radial-basis function, and backpropagation neu- 
ral networks. Massachusetts Institute of Technology. 
Neural Computation 3, 440-449. 

Miller, W.M., 1992. Classification analysis of Florida grapefruit 
based on shape parameters. In: G.K. Brown and Y. Sarig 
(Eds.), Food Processing Automation II. ASAE Publ. 02-92. 
ASAE, St. Joseph, MI, pp. 339-347. 

Miller, W.M., Drouillard, G.P., 1997. On-line blemish, color, 
and shape analysis for Florida citrus. In: M. Salius (Ed.), 
Sensors for Nondestructive Testing. Measuring the Quality 
of Fresh Fruits and Vegetables. NRAES-97. NRAES, 
Ithaca, NY, pp. 249-260. 

Moons, E., Dardenne, P., Dubois, A., Sindic, M., 1997. Non- 
destructive visible and NIR spectroscopy for the determina- 
tion of internal quality in apple. In: M. Salius (Ed.), Sensors 
for Nondestructive Testing, Measuring the Quality of Fresh 
Fruits and Vegetables. NRAES-97. NRAES, Ithaca, NY, 
pp. 122-132. 

Rehkugler, G.E., Throop, J.A., 1986. Apple sorting with ma- 
chine vision. Trans. ASAE 29 (5), 1388-1397. 

Rogers, S.K., Kabrinsky. M., 1991. An Introduction to Biolog- 
ical and Artificial Neural Networks for Pattern fees 
tion. SPIE, Bellingham, WA. 

Ruck, D.W., Rogers, S.K., Kabrinsky, M., 1989. Feature 
Selection Using a Multilayer Perception. Report of Air 
Force Inst. Of Technology. Wright-Patterson AFB, OH. 

Schalkoff, R.J., 1992. Pattern Recognition: Statistical, Struc- 
tural and Neural Approaches. Wiley, New York. 

Schatzkim Tak... Half, *RERSR Y oUns Ro Canam elec weer Ome 
Toyofuku, N., 1997. Defect detection in apples by means of 
x-ray imaging. In: M. Salius (Ed.), Sensors for Nondestruc- 
tive Testing. Measuring the Quality of Fresh Fruits and 
Vegetables. NRAES-97. NRAES, Ithaca, NY, pp. 161-171. 

Shahin, M.A., Tollner, E.W., 1997. Detection of watercore in 
apples using x-ray line scans feature extraction and classifi- 
cation, In: Sensors for Nondestructive Testing, Measuring 
the Quality of Fresh Fruits and Vegetables. NRAES-97. 
NRAES, Ithaca, NY, pp. 389-400. 

Tao, Y., Chance, L., Liu, B., 1995. Full scale fruit vision sorting 
system design-factors and considerations. In: R. Lacey 
(Ed.), Food Processing Automation ITV. ASAE Publ. 14-95. 
ASAE. St. Joseph, MI, pp. 14-22. 

Upchurch, B.L., Throop, J.A., Aneshansley, D.J., 1994. Influ- 
ence of time, bruise-type and severity on near-infrared 
reflectance from apple surfaces for automatic bruise detec- 
tion. Trans. ASAE 37 (5), 1571-1575. 

USDA, 1976. United States Standards for Grades of Apples. 
US-GPO:1987-201-037:60201/AMS. 


9 
Paper No. as 


An ASAE Meeting Presentation 


Inspection of Processed Fruit Before and After Peeling 
James A. Throop, Daniel J. Aneshansley 


Cornell University, Department of Agricultural and Biological 
Engineering, Ithaca. NY 14853 


Written for presentation at the 
1997 ASAE Annual International Meeting 
Sponsored by ASAE 


Minneapolis Convention Center 
Minneapolis. Minnesota 
August 10-14. 1997 


Reflectance spectra from apple images with rot. bruise, cracks, codling moth, leaf 
roller, scab, hail, corking, and bitter pit typical of the type of damage found on 
apples entering a processing plant were captured every 10nm for wavelengths 
between 460nm - 1030nm. The Mahalanobis distance was used to identify three 
wavelength bands (540nm, 740nm. and 1010nm) for segmentation of defects on 
apples which would normally result in trim waste. Defects segmentation was 
possible by simple image subtraction.and thresholding. This technique could be 
used for the inspection of apples entering a processing plant.. 

Meywords: Apples. Apple Quality, Defects, Defect Detection, Inspection, Image Analysis, 
| Image Processing, 


Summary: 







)1 P thi ical presentation does not necessanly reflect 
|The authoris) is solely responsible for the content of this technical presentation. The technical preser \ 
the official Spaittan of ASAE. and its pmnting and distribution does not constitute an endorsement of views which may be expressed. 


Technical presentations are not subrect to the formal peer review process by ASAE editorial committees: therefore. they are not to be 
presented as refereed publications. 


Quotation from this work should state that it is from a presentation made bv (name of author) at the (listed) ASAE meeting. 


EXAMPLE — From Author's Last Name. Initials. “Title of Presentation. ” Presented at the Date and Title of meeting, Paper No. X. 
ASAE. 2950 Niles Road. St. Joseph. MI 49085-9659 USA. 


| For information about securing permission to reprint or reproduce a technical presentation, please address inquines to ASAE. 


ASAE. 2950 Niles Rd.. St. Joseph. MI 49085-9659 USA 
Voice: 616.429.0300 FAX: 616.429.3852 E-Mail: <hq:a asae.or2> 


ASAE Paper 97-6044. Throop and Aneshansiey 


Acknowledgments: This project is supported in part by USDA Cooperative Agreement 58- 
1931-5-019 with the ARS-USDA Appalachian Fruit Research Station, Kearneysville, WV, 
and by the Agricultural Experiment Station at Cornell University with Hatch Regional Project 
NE179 and Hatch Project 412. We would also like to acknowledge the continued assistance 
and expertise of Mr. Bill Anger of ARS-USDA Appalachian Fruit Research Station in this 
project. 


CORNELL UNIVERSITY IS AN EQUAL OPPORTUNITY, AFFIRMATIVE ACTION 
EDUCATOR/EMPLOYER 


ASAE Paper 97-6044 Throop and Aneshansiey 


Abstract: 


Three cultivars of cuil apples from a processing plant were examined before and after peeling. 
Retlectance spectra were measured from images captured every 10nm for wavelengths between 
460nm and 1030nm. Typically the cull apples were rotting and little could be determined about 
the reflectance spectra of other types of damage. By visual inspection of the fruit at the 
processing plant, 9 typical defects were found which cause trim waste. There defects were 
bitter pit, bruising, codling moth, corking, cracks, hail, leaf roller, rot, and scab. Hand-picked 
apples with these types of damage were used to find the reflectance spectra for each individual 
kind of defect. Eight cultivars were included. Discriminant analysis was used to classify 
damaged from undamaged tissue. The magnitude of the Mahalanobis distance was used to 
identify the wavelengths of reflected light from the apple surface with the lowest and greatest 
contrast between damaged and undamaged tissue. Three wavelengths, (540nm, 740nm, and 
1010nm), were identitied which produced apple images that either had a reflectance nearly 
equal to undamaged areas or a retlectance much different than undamaged areas. The image at 
the wavelength with the greatest reflectance change for damaged tissue was subtracted from the 
image at the wavelength with the least reflectance change. Thresholding the resultant image 
segmented the damaged areas. 


Background: 


Unlike fresh-market apples where color, crispness, and flavor are the main factors for 
consumer acceptance. processing apples are valued for cultivar, fruit size, and quality 
characteristics that result in reduced trim waste. Ladd and Martin (1976) proposed that 
researchers use product characteristics for differentiating processed apples by cultivar, size and 
grade. This view of the product as a collection of characteristics was first proposed by 
Lancaster (1971). This view values processed apples for their utility - bearing characteristics 
where prices paid for the raw product depend on the amount of each characteristic contained in 
the product (Lucas. 1975). This has resulted in the development of pricing models through the 
estimation of hedonic price functions, which are regressions of price received as a function ot 
the product’s quality attributes (Lucas. 1975). Tronstad et. al. (1992) developed a hedonic 
price model for fresh - market apples using characteristics of location, time, cultivar, size, 
grade, and storage method as they relate to retail market pricing. 


Harper et. al., (1992) developed a model based on production - related quality attributes for 
processing apples. Three varieties of processing apples (York Imperial, Rome Beauty, and 
Golden Delicious) were assessed for size and nine quality defects including trim bruise, 
misshapen fruit, bitter pit, decay, internal breakdown, scab, hail damage, insect damage, and 
York spot. Price discounts in the sample were statistically significant for fruit size. bruising, 
bitter pit, decay, misshapen fruit, and internal breakdown. Insect damage, hail damage, apple 
scab and York spot did not cause significant price discounts because of the small amount of 
damage present. This study demonstrated that further expenditure to reduce the quality detects 
was uneconomical to the grower due to the small discount of apple price. It is obvious that 
(1) all defects can not economically be completely eliminated and (2) the processor cannot 
afford to ignore these quality defects because of the large volume of fruit purchased. Price 
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discounts for cultivar and size differences were 10 to 30 times greater than any discounts for 
the nine quality defects tested. 


At present, apples entering the processing plant are random sampled and inspected for fruit 
that are undersized and/or have defects affecting quality similar to those described above. The 
report presently returned to the producer has very limited information useful for orchard 
management that would improve apple quality. Automatic defect identification during the 
inspection process would be useful to return specific information to the producer so that 
orchard management changes could be implemented to correct for deficiencies during harvest 
and transporting to the plant. Automatic sorting would be more consistent than present human 
inspectors who gradually tire and vary in level of expertise. Aneshansley et. al.( 1997) has 
shown that specific narrow and broad bands of wavelengths of light have been identified by 
discriminant analysis to provide the greatest contrast in reflectance between damaged and 
undamaged tissue. These same techniques will be used to demonstrate that defects in 
processing apples that cause trim-waste can easily be segmented by simple fast image 
processing methods such as image subtraction and thresholding. 


Objectives: 


The objectives of this study were: 

1) to obtain reflectance spectra between 460nm and 1030nm at 10 nm intervals for 
processing apples with damage from rot, bruising, cracks, codling moth, leaf roller, scab, hail, 
corking, and bitter pit which require excessive trimming after peeling. 

2) determine if the difference in reflectance spectra of damaged areas on apples can be 
used to segment defects by selecting specific wavelength bands that produce the high and low 
contrast in the captured images between damaged and undamaged tissue. 

3) show that segmentation of damaged areas for automatic apple inspection of fruit 
entering the processing plant can be done quickly by subtracting the image with high contrast 
from the image with low contrast and simple thresholding. 


Materials and Methods: 


During the 1996 harvest season, a total of 25 cull apples from 3 cultivars (Red Delicious, 
York, and Rome Beauty) were removed from bulk apple bins in the storage yard of National 
Fruit Products, Winchester, VA. Because the cull apples with different defects from the 
processor also typically were rotting, little could be determined about the reflectance spectra 
for each different kind of damage. By visual inspection of the fruit at the processing plant, 9 
typical trim-waste causing defects were found including bitter pit, bruising, codling moth, 
corking, cracks, hail, leaf roller, rot, and scab. Apples were then gathered from the orchard 
with these specific defects to determine the spectral characteristics useful for segmentation of 
each individual kind of defect from apple images for easy image processing. 


During the 1995 and 1996 season, 18 - 22 damaged apples for each defect and including 8 
cultivars (York, Golden Delicious, Red Delicious, Stayman, Gala, Empire, Granny Smith, and 


ASAE Paper 97-6044 Throop and Aneshansiey 


Rome Beauty) were hand harvested from orchards near the USDA Appalachian Fruit Research 
Station, Kearneysville, WV. The apples were placed in common cold storage until 24 hours 
before images were captured. 


The description of image capture, light filtering, and fruit lighting can be found in Throop and 
Aneshansley, 1997. Each cull apple from the processor was imaged and then peeled (White 
Mountain Appie Parer and Corer, Winchendon, MA) and imaged again. 


The average grey level intensity of undamaged and damaged areas 4 pixels wide by 4 pixels 
high were interactively located and recorded from the apple images. Discriminant analysis of 
the two classes of tissue (damaged and undamaged) was performed (Minitab 11.0, Minitab 
Inc., State College, PA). The classification error and Mahalanobis distance between class 
means was recorded for each 10nm change in wavelength between 460nm and 1030nm. 
Whenever the classification error was the smallest tor both classes. the Mahalanobis distance 
was the greatest indicating the greatest reflectance difference between damaged and undamaged 
tissue. 


Resuits and Discussion: 


Figures | shows the Mahanlanobis distance for unpeeled York, Rome, and Red Delicious 
apples computed from images of cull fruit from the processing plant. The apples were being 
stored outside the processing plant waiting to be processed. Consequently almost every cull 
fruit had rot damage in addition to some other defect such as bitter pit, scab, bruising, cracks. 
and gashes. As evidenced by the greater Mahalanobis distance, the damaged tissue on the cull 
fruit showed the greatest difference from undamaged tissue for a wavelength band between 
700nm and 760nm. After the same cull apples were peeled, they showed a greatest contrast 
between damaged and undamaged tissue for a visible wavelength band between 500nm and 
640nm and a near - infrared (NIR) band between 960nm and 1000nm (Figure 2). This shift of 
the reflectance spectra to the visible region is a direct result of the removal of the skin as 
indicated by a shift in magnitude of the Mahalanobis distance for the visible wavelengths. 
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Figure 1. Plot of Mahalanobis Distance and Wavelength for Three Cultivars of Unpeeled Cull 
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Figure 2. Plot of Mahalanobis Distance and Wavelength for Three Cultivars of Peeled Cull 
Processed Apples. 


ASAE Paper 97-5044 Throop and Aneshansieyv 


An unpeeled York apple with a crack surrounded by rot and a large bruised area is shown in 
Figure 3. The apple image captured at 730nm (B) has a greater contrast between damaged and 
undamaged tissue compared for images at the other wavelengths shown (540nm and 1010nm). 





(A) (B) (C) 
Figure 3. Image of an Unpeeled York Apple From a Bin at the Processing Plant Captured at 
Wavelengths of 540nm (A), 730nm (B), and 1010nm (C) and All Normalized to a 
Grey Level of 25). 


Consistent with the magnitude of the Mahalanobis distance from the discriminant analysis. the 
York apple after peeling had a greater contrast between damaged and undamaged tissue for 
wavelengths ot 540nm compared to the image captured at 860nm. These differences can be 
used to segment the damaged areas by simple image subtraction and thresholding (Figure 4). 





(A) (B) (C) 
Figure 4. Image of the York Apple After Peeling Captured and Normalized at 860nm (A), 
540nm (B), and Subtracting B From A and Thresholding (C). 


Bitter Pit 


The discriminant analysis of 2 groups of Golden Delicious apples with bitter pit produced 
Mahalanobis distances for each tested wavelength as shown in Figure 5. Bitter pit is a 
physiological disease where small areas of cortical tissue at the terminals of branched vascular 
bundles near the skin surface is altered to cause death of the affected cells. As the cells die, 
moisture is lost and the spots become slightly sunken and change color trom a dark green to a 
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deep red color. As time passes the spots become dark gray or brown. The Mahalanobis 
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Figure 5. Mahalanobis Distance for Wavelengths Tested for Two Different Groups of 
Golden Delicious Apples with Bitter Pit Damage. 


distance for the Golden Delicious apples was the greatest for wavelengths between 700nm and 
740nm. This occurred when the NIR reflectance for damaged tissue was lowest compared to 
undamaged tissue. For wavelengths greater than 900nm, the reflectance for damaged tissue 
was equal to or slightly higher than undamaged tissue. For wavelengths above 960nm the 
reflectance sometimes increased for damaged tissue with respect to undamaged tissue due to 
the drying of the affected cells. 





(ean) (Ce 


Figure 6. Images of a Golden Delicious Apple with Bitter Pit Captured at 1010nm (A), 


740nm (B), and the Resulting Image by Subtracting B from A Showing the Damage 
Segmented (C). 


By subtracting two normalized images, the first taken at a center wavelength of 1010nm minus 
the second taken with a center wavelength of 740nni, it is easy to segment bitter pit damage 


from undamaged tissue on unpeeled Golden Delicious apples by simple thresholding of the 
resulting image (Figure 6). 
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Bruising 


Impact bruising severe enough to cause trim-waste was produced on Red and Golden Delicious 
apples by dropping from a height of 15 cm onto concrete. Images were captured 24 hours after 
dropping. The greatest Mahalanobis distance was for wavelengths greater than 940nm 
indicating the decrease in reflectance due to the 970nm water absorption band indicating the 
presence of fluid released from the crushed cells under the skin (Figure 7). 


Bruising was segmented from unbruised tissue by subtracting a normalized image with a center 
wavelength ot 1010nm from a reference normalized image generated from a teflon 
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Figure 7. Mahalanobis Distance Versus Wavelength for Red and Golden Delicious Apples 
Bruised by a 15cm Drop onto Concrete. 


standard which was logically ANDED with a normalized image of the same apple with a 
center wavelength of 540nm (Figure 8). When the subtraction is completed, the pixels 
representing the bruise are visible after simple thresholding. 
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(A) (B) (C) 


Figure 8. Logically ANDED image of teflon reference and 540nm image of Red Delicious 
bruised apple (A), 1010nm image, and the resulting image after subtraction of B 
from A showing the segmented bruise (C) after thresholding. 


Codling Moth 
Codling moth damage had the greatest Mahalanobis distance for images captured using 


wavelengths between 700nm and 750nm for the 3 cultivars tested, Empire, Golden Delicious. 
and Stayman (Figure 9). 
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Figure 9. Mahalanobis Distance Versus Wavelength for Golden Delicious, Empire, and 
Stayman Apples with Codling Moth Damage. 


By subtracting two normalized images, the first taken at a center wavelength of 1010nm minus 


the second taken with a center wavelength of 740nm, codling moth damage can be segmented 
from undamaged tissue on the 3 cultivars tested by simple thresholding (Figure 10). 
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(A) (B) mee) 
Figure 1{0. A Stayman apple with codling moth damage captured at 1010nm (A), 740nm (B), 
and the resulting image after subtracting B from A showing the segmented codling 
moth damage (C). 


Cork Spot 


Cork spot, sometimes called York spot is suspected to be caused by a nutritional problem 
related to an imbalance of calcium and boron. The resulting damage of spots of dead cells 
show the greatest Mahalanobis distances for wavelengths between 690nm and 740nm for 
Stayman and Red Delicious and 600nm and 640nm for York apples (Figure 11). 
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Figure 11. Mahalanobis Distance Versus Wavelength for Red Delicious, York, and Stayman 
Apples with Cork Spot (York Spot) Damage. 


By subtracting two normalized images. the first captured at a center wavelengti of 1010nm 
minus the second captured with a center wavelength of 740nm, cork spot damage can be 
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segmented from undamaged tissue on Stayman and Red Delicious apples by simple 
thresholding (Figure 12). As indicated by the lower Mahalanobis distances for York apples, 
cork spot damage segmentation for some York apples was not always successful at 740nm. 
The Mahalanobis distance measured indicates that a wavelength band with a center wavelength 
of 640nm should work better (Figure 11). 





(A) (B) (C) 
Figure 12. An Image of a Golden Delicious Apple with Cork Spot Captured at 1010nm (A), 
740nm (B), and an Image with Cork Spot Damage Segmented by Subtracting A 
from B and Thresholding.(C). 


Cracking 


Fruit cracking occurs after periods of cloudy, rainy weather when rates of evaporation and 
transpiration are very low. Cultivars such as Stayman, Wealthy, and York Imperial are more 
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Figure 13. Mahalanobis Distance Versus Wavelength for Golden Delicious and Stayman 
Apples with Crack Damage. 
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susceptible for this kind of damage. The internal flesh grows at a more rapid rate than the 
outer skin and the skin is unable to expand at this same rate thus resulting in cracking. The 
Mahalanobis distance was the greatest for wavelengths of 720nm to 760nm for the 2 groups of 
Stayman and Golden Delicious apples tested (Figure 13). Crack damage often has localized 
dehydration or rot associated with it. 


Subtracting two normalized images. the first captured at a center wavelength of 1010nm minus 
the second captured with a center wavelength of 740nm, crack damage can be segmented from 
undamaged tissue on Stayman and Golden Delicious apples by simple thresholding (Figure 
14). Note that it is the cells around the dark crack that are detected due to their dehydrated 
state. Later you will see that if rot is present, it can be detected by the same process. 





(A) (B) (C) 
Figure 14. An Image of a Stayman Apple with Crack Damage Captured at 1010nm (A), 
740nm (B), and the resulting image subtracting B from A segmenting the 
dehydrated cells around the edge of the crack. 
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Figure 15. Mahalanobis Distance Versus Wavelength for Red Delicious and Gala Apples with 
Hail Damage. 
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Hail 


Hail damage can occur anytime during the growing season. It has an appearance similar to 
cork spot with small sunken areas just under the skin of dead dehydrated cells. The 
Mahalanobis distance is the greatest for wavelengths between 700nm and 750nm (Figure 15). 
The Gala apples tested had a even greater Mahalanobis distance for wavelengths in the visible 
red region. 


By subtracting a Red Delicious normalized apple image captured with a center wavelength ot 
740nm from a normalized image with a center wavelength of 1010nm, the result is an image 
that can easily be thresholded to segment the hail damaged areas. The damage on the 1010nm 
image has a reflectance equal or greater than the undamaged tissue while the reflectance of the 
damaged areas on the 740nm image is less than undamaged tissue (Figure 16). 





(A) (B) (C) 
Figure 16. A Normalized Image of a Red Delicious Apple with Hail Damage Captured at 
1010nm (A), 740nm (B), and the Resulting Image After Subtracting B from A and 
Thresholding (C). 


Leaf Roller 


Leat roller damage is caused by the larvae consuming areas of skin and outer flesh. The 
discriminant analysis tor both Red and Golden Delicious apples produced large Mahalanobis 
distances between classes of damaged and undamaged tissue for wavelengths between 700nm 
and 820nm. Very small Mahalanobis distances were found for wavelengths beyond 960nm 
(Figure 17). Large Mahalanobis distances for Golden Delicious apples were also produced for 
wavelengths in the color region. 


By subtracting two normalized Golden Delicious images, the first taken at a center wavelength 
of 1010nm minus the second taken with a center wavelength of 740nm, leaf roller damage can 
be segmented from undamaged tissue on the 2 cultivars tested by simple thresholding (Figure 
18). 
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Figure 17. Mahalanobis Distance Versus Wavelength for Red and Golden Delicious Apples 
with Leat Roller Damage. 





(A) (B) (C) 
Figure 18. A Normalized Image of a Golden Delicious Apple with Leaf Roller Damage 
Captured at 1010nm (A), 740nm (B), and the Resulting Image After Subtracting B 
From A and Thresholding (C) 


Rot 


Rot comes in many forms and is present when organisms pass through the protective skin 
which has been damaged or through the open lenticels allowing passage of the organism into 
the moist fleshy tissue inside of the fruit. The distance between classes of damaged and 
undamaged Golden and Red Delicious apples was the greatest for wavelengths between 700nm 
and 750nm (Figure 19). The Mahalanobis distance was lowest for wavelengths beyond 960nm. 
The Golden Delicious apples also had higher Mahalanobis distances for most of the visible 
wavelengths as well. 
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If we subtract a normalized Crispin apple image captured at a center wavelength of 740nm 
from an image captured at a center wavelength of 1010nm, the resulting image can be used to 
segment the rot from undamaged tissue by simple thresholding (Figure 20). 
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Figure 19. Mahalanobis Distance Versus Wavelength for Red and Golden Delicious Apples 
with Rot Damage. 





(A) (B) (C) 
Figure 20. A Normalized Image of a Crispin Apple with Rot Damage Captured at 


1010nm (A), 740nm (B), and the Resulting Image After Subtracting B From A and 
Thresholding (C). 


ASAE Paper 97-6044 Throop and Aneshansiey 


Apple Scab 


Apple scab is caused by a fungus and can appear during cool moist conditions anywhere apples 
are grown. The fungus enters through a rupture in the cuticle and produces brown or dark 
green lesions on the surtace of the apple. For severe cases, the fruit shape can be altered 
causing poor peeling in automatic peeling machines. The Mahalanobis distance for Granny 
Smith. Red Delicious and McIntosh apples were very large for wavelengths between 700nm 
and 740nm (Figure 21). Granny Smith had Mahalanobis distances large enough for class 
separation between damaged and undamaged fruit also for most of the visible wavelengths. All 
ot the cultivars tested had reduced Mahalanobis distances for wavelengths beyond 960nm. 
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Figure 21. Mahalanobis Distance Versus Wavelength for Red Delicious, Granny Smith, and 
McIntosh Apples with Scab Damage. 





(A) (B) (C) 
Figure 22. A Normalized Image of a McIntosh Apple with Scab Damage Cartured 7t 1010nm 
(A), 740nm (B), and the Resulting Image After Subtracting B From &é and 
Thresholding (C). 
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If we subtract a normalized McIntosh apple image captured at a center wavelength of 740nm 
from an image captured at a center wavelength of 1010nm, the resulting image can be used to 
segment the rot from undamaged tissue by simple thresholding (Figure 22). 


While viewing apples at the processing plant, it became obvious that defects such as sooty 
blotch and severe flyspeck may not be ignored if automatic sorting is implemented for 
processed fruit. Figure 23 shows a Golden Delicious apple image with sooty blotch that has 
been processed as if checking for bruises. Severe sooty blotch has the potential of hiding other 
defects from a camera. By talking with the processors, we learned that they are becoming 





(A) B) © 
Figure 23. Logically ANDED Image of Teflon Reference and 540nm Image of a Golden 
Delicious Apple With Sooty Blotch (A), 1010nm Image, and the Resulting 
Image After Subtraction of B From A Showing the Segmented Sooty Blotch (C). 


concerned with this surface defect which is peeled away because they are presently processing 
apple skins and cores to obtain apple cider vinegar and feel that the sooty blotch fungus, 
Gloeodes pomigena, could contaminate this product. 


Conclusions: 


Images of unpeeled processing apples with damage from rot, bruising, cracks, codling moth, 
leaf roller, scab, hail, corking, and bitter pit requiring excessive trimming after peeling were 
successfully processed segmenting damaged tissue from undamaged tissue. Three wavelengths 
(540nm, 740nm, 1010nm) were found that simplified image processing to simple image 
subtraction and thresholding for unpeeled apples. In most cases an image at one of the three 
wavelengths had high contrast between damaged and undamaged tissue while an image at one 
of the other two wavelengths had low contrast. By subtraction of the high contrast image from 
the low contrast image and thresholding, segmentation of the damaged areas was completed. 
York spot on the York cultivar was not always segmented successfully at 740nm and the 
Mahalanobis distance measured indicated that a wavelength band with a center wavelength of 
640nm should work better. 


Unlike unpeeled apples, peeled apples had the greatest contrast between damaged and 
undamaged tissue for the visible wavelengths 500nm to 640nm and a near - infrared (NIR) 
band between 960nm and 1000nm as indicated by the magnitude of the Mahalanobis distance. 
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Sooty blotch was found to block the reflectance changes for some defects like bruising, but the 
processing industry indicated that better control of this surface defect is desired. 
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Summary: Reflectance spectra from 22 defects commonly found on 10 different apple 

, cultivars were measured and processed to identify 3 wavelength bands of 50 - 
60nm in width for defect segmentation. A combination of image processing 

| techniques and images of the same apple captured at these different wavelengths 

were described to accomplish the segmentation process. 
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Abstract: 


The reflectance spectra for 22 defects found on 10 different apple cultivars was measured 
every 10nm for wavelengths between 460nm to 1030nm. Discriminant analysis was used to 
identify three wavelength regions, 540nm, 740nm, and 1010nm, which produced images with 
the highest and lowest contrast for damaged tissue with respect to undamaged tissue as 
indicated by the highest or lowest Mahalanobis distance respectively. Subtraction of an apple 
image with the defect having the highest contrast compared to undamaged tissue from an 
image of the same apple with little difference between damaged and undamaged tissue and 
thresholding easily segmented bitter pit, botryosphaeria rot, chemical. codling moth, corking, 
cracking, hail, leaf roller, rot, and scab damage. 


Bruising, scald, punctures, sooty blotch, and fly speck had a lower reflectance for all near- 
infrared wavelengths. The lowest reflectance compared to undamaged tissue occurred for 
wavelengths between 960nm and 1030nm. Images captured with these defects were segmented 
by subtracting these images from a teflon reference image with intensity values approximating 
the values of undamaged tissue. 


Detects such as russet, early frost damage, insect stings, blister spot. and powdery mildew are 
only skin deep and affect skin color. These defects appeared brighter on red apples and darker 
on yellow/green apples. This caused the defects to be undetectable on red/green varieties such 
as Empire. For monochoric apples such as Red or Golden Delicious, these defects were 
segmented by subtracting an image captured at 540nm from an image captured at 1010nm. The 
defect on the resulting image for yellow/green apples was segmented by simple thresholding. 
For red colored apples, the resulting image must be subtracted from a teflon reference image 
before thresholding. Blister spot was segmented in the same manner but a Sobel edge detector 
was applied to the resulting image before thresholding. 


Sunburn was segmented on Red Delicious apples by subtraction of an image captured at 
1010nm from an image captured at 540nm and thresholding. 


Background: 


The quality of apples in the retail market are greatly influenced by maturity at harvest, 
handling, picking, transportation, and storage. Dealers judge apple quality by physical features 
such as size, shape, color, appearance, and finger feel. Consumers decide to purchase based on 
eating qualities of the apple although quite often at the time of purchase only external 
appearance is available (S. Dhanaraj et. al. 1980). Although studies have shown that 
appearance has little to do with eating quality, it certainly plays a major part in the final 
decision to purchase (E. Hohn, 1990). The “juicy, crisp, and flavorful” part of the consumer’s 
decision come from memory of the last time an apple was consumed. 


Appearance is the major criteria for quality sorting of apples for consumers. Typical reasons 
for culling apples are shown in Table 1. Although many of the defects are from insect and 
disease, the largest number of fruit culled resulted from size and color (W. D. Gerling. 1983). 
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Table 1. Typical Factors Causing Culling of McIntosh Apples During the 1992 & 1993 
Growing Seasons in the Hudson Valley, Highland. NY. 


Factor % of Sampled Fruit % of Total Culls 
a A ONE DRRREDO RL OF SISA Os Ee) ee ee ee eee 
Undersize a5 38 
Color 6.9 35.0 
Bruise OFZ Dh 
Mechanical Oe 4.9 
Stem Puncture 0.1 130) 

Hail 0.8 9.0 
Malformed 0.3 iS 
Finish 08 2H 
Disease ee 0.7 
Insect Damage On Ze) 
Total 9 100.0 


Table 2. Average Percentage of Defective Apples According to Origin or Time of Occurrence 
of the Grade Defect. 


Variety Orchard Harvest & Handling Storage All Sources 
Delicious 20.3 9.1 ie 28.1 
Jonathan 220 1337 4.6 35.8 
McIntosh 19.2 21.0 0.1 35a) 
Rome S14 17.9 1.9 41.2 
All Varieties Z23r8 13.4 ee 35.8 


Dewey et al. 1971 examined 18 lots of 6 cultivars of stored apples to determine the probable 
cause and origin of defects found on the apples and classified them into the following 
categories: (a) growing conditions, (b) harvest and handling conditions, and (c) storage or 
dumping operations (Table 2). Preharvest defects included small size, inadequate color, limb- 
rub, russeting, hail damage, and scab. The most serious defect during harvesting was bruising. 
Defects from storage included mostly rot, decay, and internal breakdown. 
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Dewey et. al. 1971 found that the number of 42-lb. bushels of orchard run fruit necessary to 
_ packout 40-lb. units of U. S. Fancy Michigan apples increased as the number of defects 
_ increased causing lower packout rates (Table 3). If the packout rate decreased from 95% to 
35%, the storage, sorting and packing costs nearly tripled. Two chances for increasing profit 
were found: (1) improve fruit quality during production and harvesting, (2) do not store low 
grade fruit. If high speed automatic sorting was available. more apples could be sorted before 
storage to prevent storage of low grade fruit. 


Table 3. The Effect of Packout Rate on the Number of 42-lb Orchard Run Bushels Required 
to Filla U. S Fancy 40-lb. Unit. 


Packout of Orchard Run Apples Nieldor UsS > Fancy 
(Percent) (Units per Orchard Bushel) 
95 1.00 
85 ele 
HD aoa! 
65 1.46 
55 Leds 
45 Dephd 
35 2. od 


In spite of the fact that defect sorting represents a loss of product value, quality sorting of 
apples is necessary to provide the consumer with a quality apple with excellent appearance or 
other truits wiil replace appies in the retail market. 


Assessing apple quality. doth for maturity and defects, by spectral reflectance dates back to the 
early 1940’s (Lott, 1943). The recording of the reflectance from the apple surface of different 
wavelengths of light resulted in identifying particular wavelengths to define maturity (Bittner, 
et. al. 1968, bruising (Ingle and Hyde, 1968, Brown et. al., 1974, Fuzzen, 1981, Upchurch 
et. al. 1990), and other defects (Reid, 1976, Rehkugler et. al. 1989). 


Aneshansley et. al. 1997 captured images with a video camera with a tunable bandpass filter in 
front of the camera lens. Images of the reflectance for center wavelengths between 460nm and 
1030nm were captured in 10 nm intervals for 5 cultivars and 18 defects. Discriminant analysis 
was performed on the data and the classification error between damaged and undamaged tissue 
and the Mahalanobis distance for each defect were used to statistically identify the wavelengths 
which produced the greatest reflectance contrast between damaged and undamaged tissue. This 
paper is a continuation of this work. 
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Objectives: 


Expand the present data base of images of the reflectance spectra of defects for additional 
cultivars and through discriminant analysis, define specific wavelength bands to insure 
maximum and minimum contrast in reflectance between damaged and undamaged tissue for 
each defect and determine if these wavelength bands work for different cultivars with the same 
detect. 


Determine which defects can be quickly segmented by subtracting 2 images, one captured 
using a wavelength band that produces the highest contrast between damaged and undamaged 
tissue, and a second image captured that produces the lowest contrast between damaged and 
undamaged tissue. After the subtraction process, determine which defects can be segmented 
with simple thresholding. 


If subtraction of two images and thresholding fails to segment the defect, attempt to define 
another image processing technique for segmentation. 


Materials and Methods: 


During the 1995 and 1996 harvest season, 18 - 22 damaged apples for 22 defects from 10 
cultivars (York, Golden Delicious. Red Delicious, Stayman, Gala, Empire, Granny Smith, 
McIntosh, Crispin, and Rome Beauty) were harvested from orchards near the USDA 
Appalachian Fruit Research Station, Kearneysville, WV. The apples were held in common 
cold storage until 24 hours before images were captured (Table 4). 


Two camera/lens/adjustable filter combinations were used. System 1 consisted of a CCD 
monochrome camera (Model 4900. Cohu Electronics, San Diego, CA) with its gain adjusted to 
full and equipped with a 25mm lens with the aperture set at t1.4. A tunable filter system 
(Model: VS-VIS3-20-MC-20-SQ, Cambridge Research & Instrumentation Inc., Cambridge, 
MA) was placed in front of the camera lens. The filter had variable attenuation control and 
was adjusted for 29 different wavelengths between 460nm to 750nm, both of which were 
controlled from the serial port of the image capture computer. For each wavelength selected 
the filter had a 20nm bandpass and a 20nm aperture diameter. System 2 consisted of a CCD 
monochrome camera (Model 4800, Cohu Electronics, San Diego, CA) equipped with a 25 mm 
lens with the aperture set at f1.4. A tunable filter system (Model: VS-NIR1-35-MC-20-RN, 
Cambridge Research & Instrumentation Inc., Cambridge, MA) was placed in front of the 
camera lens. The filter had variable attenuation control and was adjusted for 29 different 
wavelengths between 750nm to 1030nm, both of which were controlled from the serial port of 
the image capture computer. For each wavelength selected the filter had a 35nm bandpass and 
a 20nm diameter aperture. A white teflon cylinder 76mm diameter by 76mm high was placed 
in the field of view of both systems. For each wavelength tested, the correct attenuation setting 
for the neutral density portion of the variable filter allowing a camera output of 175 out 255 
grey levels was recorded in a look up table. This assured that the camera output was 
normalized to a standard output grey level while viewing the teflon cylinder regardless of 
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Table 4. Table Showing the Defects Tested on the Left, Cultivars Tested on the Right. and the 
Cultivars Which Had the Defect in the Center. 


Detect Cultivars Tested Cultivars 

( By Letter) 
Bitter Pit A,B A. Red Delicious 
Blister Spot Bae B. Golden Delicious 
Botryosphaeria Rot B C. Crispin 
Bruise A,B D. Gala 
Chemical B E. Empire 
Codling Moth Aw sie. F. Stayman 
Corking BryaG Ce LOLK 
Cracking Bor H. McIntosh 
Fly Speck AED eat I. Granny Smith 
Frost (early) Bern J. Rome Beauty 
Hail AD) 
Leat Roller A,B 
Powdery Mildew A, 
Puncture A® Bosh. 
Rot ABC 
Russet AnB-~ Eck 3H 
Scab Aba ttl 
Scald A 
Sooty Blotch ASSBab 
Sting AnBaC 
Sunburn A 


changes in camera response due to wavelength or of changes in the transmission characteristics 
of the variable filter. System 1 and system 2 were each controlled by different PCI bus Alpha 
275 MHz processor equipped computers (Model SYS-AXP275AN, BTG Micro Systems, 
Fairfax. WA and Model Cobra Cab 275, Carrera Computers Inc., Laguna Hills. CA) with 
frame grabber cards (Raptor-PCI, BitFlow, Woburn, MA) installed for image capture. Filter 
control was through the serial port via a subroutine running under image capture and 
processing software (Quantim, ZEDEC Technologies Inc., Burlington, NC). For every apple 
tested. this combination provided sequential filter adjustment and image capture for 58 images 
continuously for each of the different wavelengths tested. Images 256 pixels high by 256 pixels 
wide were captured in this study and were stored on a rewriteable 650 MB/disk PD.’\CD-ROM 
drive (Model:_LF-1004AB, Panasonic Communications & Systems Co., Secaucus. *'J). 


Diffuse lighting was produced by placing the apples inside of a 222mm diameter cylindrical 
white acrylic diffuser 406mm high with four tungsten lamps placed around the o:ss:ce. The 40 
watt lamps (model 40A. 40W, General Electric, Cleveland, Ohio) were place in a 305mm 
per side square centered about the outside of the diffuser at a height of 184mm. To minimize 
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60 Hz fluctuations. the adiustable 120 VAC to the lamps was converted to adjustable 160 VDC 
(4V pk-pk ripple) by using a 10A bridge rectifier (GBPC- 1004, General Instruments) and a 
capacitor filter (80 uf 450 VDC. 36DX-800F450DC, Sprague). 


The average grey level intensity of undamaged and damaged areas 4 pixels wide by 4 pixels 
high were interactively located and recorded from the apple images. Discriminant analysis of 
the two classes of tissue (damaged and undamaged) was performed (Minitab 11.0, Minitab 
Inc., State College, PA). The classification error and Mahalanobis distance between class 
means were recorded for each 10nm change in center wavelength between 460nm and 
1030nm. Whenever the classification error was the smallest for both classes. the Mahalanobis 
distance was the greatest indicating the greatest reflectance difference between damaged and 
undamaged tissue. 


Results and Discussion: 


Figure 1 is a typical grapn showing the variation in Mahalanobis distance as the center 
wavelength changed from 460nm to 1030nm for Empires, Golden Delicious, and Stayman 
apples with codling moth damage. The greatest Mahalanobis distance occurred at 720nm - 
740nm and the lowest at wavelengths above 950nm for all 3 cultivars shown. Further graphs 
of the Mahalanobis distance for other defects on different cultivars can be seen in Throop et. 
al. 1997 and Aneshansley et. al. 1997. 


As a result of the discriminant analysis, 3 wavelength bands centered at 540nm, 740nm, and 
1010nm were identified having either the highest or the lowest Mahalanobis distance, 
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Figure 1. Mahalanobis Distance for Every 10nm Change in Center Wavelength for Empire, 
Golden Delicious , and Stayman Apples with Codling Moth Damage. 
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depending on the physical changes caused by the defect to the apple surtace. Table 5 shows the 
defects with a solid line indicating the wavelength band with the highest Mahalanobis distances 
and the greatest contrast between damaged and undamaged tissue. If a dotted line is shown, 
this wavelength band had the lowest Mahalanobis distances indicating very low contrast 
between damaged and undamaged tissue. For defects like bruising, scald. fly speck, sooty 
blotch. and puncture damage. one wavelength band showed the greatest contrast tor 
undamaged tissue but did not have a second wavelength band where the defect had a 
reflectance similar to undamaged tissue. 


It was observed that the Mahalanobis distance remained nearly at the same level for adjacent 
wavelengths each side ot the center wavelengths of 540nm, 740nm and 1010nm. Considering 
that wider wavelength bands would supply more light to the camera allowing more exposure 
flexibility during actual image capture on the grading line. 5 images were averaged together 
before the segmentation process was started. Apple images captured at a center wavelength of 
520nm. 530nm. 540nm. 550nm. and 560nm and a bandwidth of 20nm were averaged 


Table 5. Table of Defects Indicating by Solid Lines the Wavelength Band with Greatest 
Contrast and with Dotted Lines the Lowest Contrast Between Areas of Damaged and 
Undamaged Tissue. 


Detect Center Wavelength 

540nm 740nm 1010nm 
Bitter Pit 
Blister Spot 
Botryosphaeria Rot 
Bruise 
Chemical 
Codling Moth 
Corking 
Cracking 
Fly Speck 
Frost (early) 
Hail 
Leat Roller 
Powdery Mildew 
Puncture 
Rot 
Russet 
Scab 
Scald 
Sooty Blotch 
Sting 
Sunburn 
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Blister Spot on Golden Delicious 


Figure 2. A Golden Delicious Image Captured at 540nm,. 1010nm, and the Resulting Image 
After Subtracting the 540nm From the 1010nm Image, Edge Detection with a Sobel 
Filter, and Low Pass Filter Using a 7 x 7 Kernel. 


together to form a single image with a center wavelength of 540nm and an approximate band 
width of 50nm. Images captured at a center wavelength of 720nm, 730nm, 740nm, 750nm, 
and 76Onm and a bandwidth of 35nm were averaged together to form a single image with a 
center wavelength of 740nm and an approximate bandwidth of 60nm. Similarly, images 
captured at a center wavelength of 990nm, 1000nm, 1010nm, 1020nm. 1030nm and a 
bandwidth of 35nm were averaged together to form an image with a center wavelength of 


1010nm and an approximate bandwidth of 60nm. All images were .:ormalized to an intensity 
of 220 grey levels: 


Figure 2 shows a Golden Delicious apple image with blister spot captured at a center 
wavelength of 540nm (high defect contrast) and a second image captured at a center 
wavelength of 1010nm (low defect contrast). For every image where the defect has high 
contrast and low contrast with undamaged tissue, it is possible after 1mage normalization to 
subtract the high defect contrast image from the low defect contrast image and apply simple 
thresholding to the resulting image to segment the defect. Blister spot segmentation starts by 
the subtraction of an image captured at 540nm from an image captured at 1010nm. but 
requires the resulting image to be filtered by a Sobel edge detector aiid !ow pass filter to 
remove the lenticels. The size of kernel used in the low pass filter is czpendent on the size of 
blister spots and lenticels (Figure 2). For this group of apples a 7 x 7 kernel size was used. 


Bitter pit. botryosphaeria rot, chemical, codling moth, corking, crac.:ing, hail, leaf roller, rot, 
and scab damage was segmented by subtracting normalized apple images captured at 740nm 
from images captured at 1010nm and simple thresholding (Figure 3). These defects have a 
lower intensity value for the defective area at 740nm and an equal or higher reflectance than 
undamaged tissue for images captured at 1010nm. This wavelength, 1010nm, is near a water 
absorption band (960nm) in the near-infrared and the loss of internal cell fluid due to the 
defect is indicated by the higher reflectance than for undamaged tissue. 


Bruising, scald, punctures, sooty blotch, and fly speck had grey levels different from 
undamaged tissue for all of the wavelengths tested. This was also indicated by the Mahalanobis 
distance which remained high for all near-infrared wavelengths tested with the highest values 
for wavelengths beyond 960nm. A reference image representing the general background of 
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Cracking on ‘Stayman 


Bitter Pit on Red Delicious 





Botryosphaeria Rot on Golden Delicious 





Chemical Spray on Golden Delicious Leaf Roller on Golden Delicious 





Codling Moth on Crispin Rot on Crispin 





Corking on Golden Delicious Scab on McIntosh 


Figure 3. Images of Apples of Different Cultivars with Different Defects, the First in Each 
Series of 3 Views is a Low Contrast Image of the Defect Captured at 1010nm, the 
Second a High Contrast Image of the Defect Captured at 740nm, and the Third a 
View of the Segmented Defect After Subtraction of View 2 From View | and 
Thresholding. 
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undamaged tissue was created by capturing and normalizing an image of a teflon cylinder at a 
wavelength of 1010nm. Because the cylinder was a fixed size and the apples tested were ot 
variable size, the image of the teflon cylinder was logically AND with the apple image 
captured at 540nm. The resulting image is now the same size as the apple being tested and has 
grey level values very similar to undamaged tissue. By subtracting apple images captured at 
1010nm from this reference image and thresholding, bruising, punctures, scald, sooty blotch, 
and fly speck were segmented (Figure 4 ). 






Bruise on Golden Delicious Sooty Blotch on Golden Delicious 





Scald on Red Delicious 


Fly Speck on Red Delicious 





Puncture on McIntosh 


Figure 4. Images of Apples of Different Cultivars with Different Defects. -he First in Each 
Series of 3 Views is a Background Reference Image Captured of a Teflon Cylinder 
at 1010nm and Logically AND with an Image Captured at 540nm, the Second a 
High Contrast Image of the Defect Captured at 1010nm, and the Third a View of the 
Segmented Defect After Subtraction of View 2 From View 1 and Thresholding. 
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ANS in (B) tea) 


Figure 5. Images Captured at 540nm of Russet Damage on Golden Delicious (A), Red 
Delicious (B). and Empire (C) Apples. 


Defect segmentation discussed so far has been quite successful because the defects have been 
of higher or lower contrast than undamaged tissue in the near - infrared wavelengths and 
variation in fruit color has not been a factor. Russet and defects which appear similar to russet 
such as early frost damage, insect stings, and powdery mildew are difficult to segment. The 
defect russet is only skin deep and mainly affects the skin color. For images captured at 
540nm, the defect looks brighter than undamaged tissue on red apples while on yellow/green 
varieties the same defect appears darker than undamaged tissue (Figure 5). For images 
captured at 740nm and 1010nm, russet appears similar to undamaged tissue. The fact that the 
damage appears brighter on yellow/green apples and darker on red apples makes this defect 
difficult to segment on varieties such as Empires which can be both red and green on adjacent 
surfaces of the apple cheek (Figure 5). 


Early insect stings not only may have a russet like appearance, but also can have local shape 
deformation as well. Figure 6 shows early insect stings with little shape deformation on Red 
Delicious and Golden Delicious apples but the Crispin apple not only has a patch of russet like 
cells visible but also is deformed with some pitting in the affected region. The russet areas are 
not visible in the images captured at 740nm but the pitted areas appear as dark spots. 


Segmentation of russet like damage is possible from images captured of monochroic apples. 
Figure 7 shows segmentation of russet on both Golden and Red Delicious cultivars. The initial 
image processing for the red and yellow/green colored apples is the same at the start where an 
image captured at 540nm is subtracted from an image captured at 1010nm. The resulting 
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(A) (B) i. 
Figure 6. Images of Early Insect Stings Captured at 540nm on Red Delicious (A), Golden 
Delicious (B), and Crispin (C). 


image is multiplied by a constant and simple thresholding is applied for russet segmentation on 
vellow/green apples. To segment russet like damage on red colored apples. the resulting image 
must be subtracted from the teflon reference image described earlier for bruising and simple 
thresholding applied (Figure 7). The same image process is used for early frost damage and 
early sting damage for defect segmentation (Figure 7). 








i 
Powdery Mildew on Red Delicious 








Sting on Red Delicious 
Figure 7. Examples of Segmentation of Russet, Insect Sting, Early Fres: Camage and 
Powdery Mildew on Monochromatic Apples. 
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Sunburn affects the skin color and can be detected in images captured at 540nm. A red apple 
image captured at 1010nm was subtracted from an image captured at 540nm. The resulting 
image was multiplied by a constant and logically AND with the 540nm image (Figure 8). 





Sunburn on Red Delicious 


Figure 8. A Red Delicious Apple with Sunburn Damage Captured at 540nm, 1010nm and the 
Resulting Image After Logically AND with the 540nm Image. 


Conclusions: 


Scab. bitter pit. botryosphaeria rot, chemical spray, codling moth. corking, cracking, hail, leaf 
roller. and rot damage were segmented from undamaged tissue by subtracting a normalized 
image captured at 740nm from a normalized image captured at 1010nm and simple 
thresholding. Type of cultivar is not a factor for segmentation. 


Blister spot damage can be segmented by subtracting a normalized 540nm image from a 
normalized 1010nm and filtering with a Sobel and low pass filter kernel. This defect was only 
tested on green/yellow cultivars. 


Bruising, scald, punctures, sooty blotch, and fly speck damage can be segmented by 
subtracting a normalized image captured at 1010nm from a reference image. The reference 
image is formed by logically AND ot a normalized 540nm image with an image captured at 
1010nm of a tetlon cylinder. Type of cultivar is not a factor for segmentation. 


Russet. early sting, early frost, and powdery mildew damage can be segmented on monochroic 
apple images captured at 540nm which are subtracted from a normalized image captured at 
1010nm. Yellow/green apples are then segmented by simple thresholding. Red apples are 
subtracted from the teflon reference image described for bruise segmentation before simple 
thresholding is applied. Segmentation was only possible on monochroic apples. 


Sunburn damage on red apples was segmented by subtracting a normalized image captured at 
1010nm from a normalized image captured at 540nm and logically AND of the resulting image 
with the image captured at 540nm. Only had Red Delicious apples with sunburn gam-ce but 
suspect that monochroic apples are required for segmentation. 
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Abstract 


A technique using body transmittance in the 450- to 1050-nm region was evaluated as a non-destructive technique 
for identifying apples with internal breakdown. Apples with internal breakdown absorbed shorter wavelengths of light 
(<750 nm) more than good apples; however, light at longer wavelengths (>750 nm) was absorbed more by good apples 
than defective ones. A classifier based on the ratio between the light intensity at 720 and 810 nm was used to segregate 
apples with internal breakdown from good apples. Only 6.3% of the good apples were incorrectly classified: however, 
bruises on the apples contributed to the incorrect classification of good apples. None of the apples with severe internal 
breakdown were incorrectly classified. Apples with very slight internal breakdown contributed to most of the 12.0% 
error of misclassifying defective fruit as good. The number of misclassified apples was reduced by adjusting the cut-off; 


however, the number of good fruit that would be discarded increased. 


Keywords: Light transmittance; Postharvest quality; Apple: Internal browning; Inspection 


1. Introduction 


Accurate non-destructive techniques for assessing 
the postharvest quality of fruits and vegetables would 
be an asset to the industry. Correct decisions about 
the quality of each item at harvest could reduce the 
postharvest losses by eliminating the storage and 
transportation of inferior quality produce. By identi- 
fying fruit that are past the optimum stage for long- 
term storage, these fruit can be marketed early prior 
to a degradation in quality. A fast non-destructive 
technique would provide an objective measurement 
of each fruit; therefore, each fruit would be of similar 
quality when it reached the consumer. 


* Corresponding author. 


Internal defects, such as watercore, brown heart, 
internal browning, internal breakdown, or insect in- 
festation, result in automatic rejection of the apple 
for fresh market. To identify the presence of internal 
defects, current practices involve a destructive test 
on arandom sample. Each apple in the sample is cut 
in half and visually inspected. If an internal defect 
is present in a certain percentage of the samples, the 
complete ‘lot’ is rejected. To date, non-destructive 
technologies for inspecting the internal condition of 
each fruit either do not exist or are very limited in 
functionality and affordability. 

A challenge to inspecting for internal defects is 
identifying a sensing technique that yields a de- 
tectable signal after penetrating deep into the product. 
Chen et al. (1989) used magnetic resonance imaging 
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to obtain images of bruises on apples, peaches, pears, 
and onion, pits in olives and prunes, and worm dam- 
age in pears. Although magnetic resonance imaging 
exhibited positive results, it is costly and slow. Tech- 
niques, such as 1-dimensional profiles, are being de- 
veloped to reduce processing time and costs (Zion et 
al., 1993). X-rays have been successfully used to de- 
tect split pits in peaches (Bowers and Dodd, 1988) 
and watercore in apples (Tollner et al., 1992). 

Light transmittance measurements were used to 
measure internal defects in fruits and vegetables. 
Birth and Olsen (1964) and Throop et al. (1989) 
reported light transmission measurements for distin- 
suishing watercored apples from good ones. Using 
a machine vision system to measure the light dis- 
tribution through the apple, Throop et al. (1989) 
reduced the errors of misclassification caused by 
high light levels around the stem. The success of this 
technique was dependent upon the sensitivity of the 
camera (Throop et al., 1994) and the length of time 
the apples had been in storage prior to inspection 
(Upchurch et al., 1994). 

Various factors contribute to the development of 
internal breakdown in apples. This is a term used to 
describe an internal disorder that is characterized by 
the browning and softening of the cortex tissue. The 
failure of the cell membranes to retain phenolic pre- 
cursors of browning within the vacuole is the cause 
of browning related to internal breakdown (Porritt 
et al., 1982). However, the mechanisms behind the 
cellular disorganization is not fully understood. The 
accumulation of sorbitol or volatiles is a contributing 
factor. Accumulation of sorbitol around the vas- 
cular bundles is evident in apples with watercore; 
therefore, watercore apples have a higher probability 
of developing internal breakdown than good apples 
(Myers, 1983). High levels of carbon dioxide during 
storage caused internal breakdown in apples during 
the 1993 season (Anonymous, 1994; Warner, 1994). 

Light transmittance has been the only technique 
reported for measuring the internal breakdown in 
fruits and vegetables. An optical density difference 
between 710 and 800 nm successfully detected inter- 
nal browning in potatoes (Birth, 1960). Differences 
in the light transmittance at 600 and 740 nm were 
used to detect internal browning in apples; however, 
the technique was unable to detect apples with slight 
browning (Fukuda and Kubota, 1979). 


Marketing apples with internal breakdown can 
negatively affect consumer acceptance. By inspect- 
ing each apple for this defect prior to shipping, only 
apples with internal breakdown would be rejected. 
The objective of this research was to evaluate an in- 
teractance measurement that could be used to detect 
the presence of internal breakdown in apples. 


2. Materials and methods 


‘Delicious’ apples were hand harvested in late 
October during the 1994 season from the orchards 
at the Appalachian Fruit Research Station, Kear- 
neysville, WV. Apples harvested late in the season 
had a high probability of having watercore at har- 
vest and developing internal breakdown during stor- 
age. These apples were stored at O°C for 6 months 
which provided ample time for development of in- 
ternal breakdown. Interactance measurements were 
recorded for 556 apples. A set of 136 apples were 
used as a calibration set. After developing the clas- 
sifier, interactance measurements were recorded on 
420 additional apples, and the performance of the 
classifier was evaluated. Interactance for each apple 
was measured with the apparatus shown in Fig. 1. 

A DC light source and computer-controlled spec- 
trograph provided the necessary instrumentation for 
recording the spectral content of the light traveling 
through the apple. The calyx-end of the apple was 
exposed to the light source by placing each apple 
on top of a 15 x 15 x 17 cm light box. To re- 
duce the amount of light leaking around the sides 
of the fruit, a 7.6-cm diameter black sponge with an 
inverted conical surface was mounted between the 
light box and calyx of the apple. The calyx-end was 
illuminated through a 3.2-cm diameter opening at 
the top of the light box and sponge seal. A 160 VDC 
source was supplied to the envelope-type tungsten 
lamp in the light box. This DC voltage eliminated 
the 60 Hz fluctuations of AC powered light sources 
and was generated by converting 120 VAC to 160 
VDC (4 V pk-pk ripple) by using a 10-A bridge 
rectifier (GBPC-1005, General Instruments) and an 
80-uf 450 VDC capacitor filter (36DX-800F450DC, 
Sprague). 

A fiber optic probe collected the energy exiting 
through the cheek of each apple. After placing the 
apple on the light box so that the light entered the 
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Fig. 1. Apparatus for measuring the spectral composition of the light transmitted through an apple. 


calyx end, a 4-mm diameter fiber optic probe was 
placed in contact with the cheek midway between 
the stem and calyx ends. The site at which the probe 
contacted the apple was marked on the peel. Only 
one site was Selected for each apple. 

Spectral composition of the light transmitted 
through the apple was measured in the 450- to 
1050-nm region. The other end of the probe was 
connected to the entrance port on a spectrograph 
(82-498, Jarrell-Ash). A 150 grooves/mm ruled grat- 
ing (11-086 985-30-49-40, Jarrell-Ash) in the spec- 
trograph yielded the required wavelength range. The 
spectral composition of the light exiting the spec- 
trograph was measured with a silicon diode array 
(TN-6112, Tracor Northern). Extremely low light 
levels required a 500-jm entrance slit in the spec- 
trograph and a 5-s exposure time for the diode array. 
This combination of grating and slit yielded a spec- 
tral resolution of 12 nm. The signal from the diode 
array was digitalized by a photodiode array acquisi- 
tion system (TN-6600, Tracor Northern) with 14-bit 
resolution. 

Each apple spectrum was converted to a percent 
transmittance prior to analysis. Before measuring the 
spectra of the apples, the transmittance through an 
80-mm diameter Teflon cylinder was recorded as a 
reference. This measurement was only made once 
a day. The cylinder was centered on the light box, 
and the probe mounted 25 mm above the base of 
the cylinder for the measurement. A percent trans- 


mittance was calculated by dividing each apple spec- 
trum by the reference spectrum. 

A visual score that was indicative of the internal 
condition was assigned to each apple. The amount 
of browning was used as a measure of the internal 
breakdown. After measuring all apples, each fruit 
was sliced perpendicular to the stem calyx axis. A 
subjective score of (0) no, (1) slight, (2) moder- 
ate, or (3) severe browning was assigned to each 
apple. When the region of browning did not ex- 
tend completely around the cut half, the location of 
the browning relative to the measurement site was 
recorded. For apples with no signs of browning, the 
calyx half of the apple was cut parallel to the core. 
This cut was essential to insure that no browning 
occurred around the calyx. If browning was present, 
a new score of slight browning was assigned. A 
classifier was developed to segregate the apples with 
internal breakdown (scores 1—3) from non-defective 
apples. 


3. Results and discussion 


Browning of the tissue within an apple affected 
the spectral content of the light transmitted through 
the fruit. The average transmission spectrum for each 
class of the calibration set is shown in Fig. 2. Gen- 
erally, absorption at wavelengths between 720 and 
750 nm was greater for apples with internal brown- 
ing than apples without the defect. At wavelengths 
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Fig. 2. Average spectral composition of the light transmitted 
through apples with and without internal browning. 


above 750 nm, the transmittance was higher for de- 
fective apples. The degree of absorption by the apple 
was dependent upon the amount of browning present 
in the tissue. 

A classifier for identifying apples with internal 
breakdown was developed based on the spectral 
characteristics of the transmittance. From Fig. 2, 
the amplitude of the transmittance at several wave- 
lengths was dependent upon the internal condition of 
the apple. To minimize the influence of path length, 
a ratio between the transmittance at 720 and 810 
nm was selected as the feature for discriminating 
between defective and good apples. The ratio de- 
creased as the amount of browning increased and 
was correlated (~ = 0.71) with the degree of inter- 
nal breakdown. Analysis of variance and subsequent 
mean separation with confidence intervals of the data 
(F = 221.2, P < 0.0001, ANOVA, SAS/LAB, SAS 
Institute, Inc., SAS, 1992) revealed three distinct 
classes, no browning, slight, and moderate/severe 
browning (Table 1). A lack of difference between 
the moderate and severe browning class could be 
attributed to the assignment of the subjective score. 

A classifier based on the transmission ratio was 
developed for identifying apples with internal break- 
down. Using the 95% confidence intervals for the 
class means (Table 1), apples with a ratio greater 
than or equal to one were classified as good ap- 
ples, while any apple with a ratio less than one 
was classified as defective. The ratio feature was the 
performance of the classifier based on the ratio is 


Table 1 

Confidence intervals for the four classes of apples with and 
without internal browning. Scores represent the degree of internal 
browning (0, none: 1, slight; 2. moderate: and 3, severe) assigned 
to the calibration set 


Score Number of samples 95% Confidence interval * 
0 28 1.24: 1.36 (a) 
1 36 0.74: 0.85 (b) 
2 53 0.49: 0.58 (c) 
3 19 0.36: 0.51 (c) 


“Confidence intervals with a different letter are significantly 
different at the 95% confidence level. 


Table 2 

Classification errors associated with separating apples into two 
classes. no internal browning and internal browning, using a 
feature based on the ratio between the transmittance at 710 and 
820 nm. An apple with a ratio equal to or greater than 1.0 was 
classified as a good apple, while any apple with a ratio less than 
1.0 was classified as defective 


Predicted good Predicted brown 


Actual good 237) 16 
Actual brown 20 147 


shown in Table 2. Since any amount of browning 
would result in rejection of the apple, all of the 
apples with internal browning were combined into 
a single class. No effort was made to distinguish 
between the classes of browning. Only 16 out of the 
253 (6.3%) good apples were misclassified as having 
browning, while 20 out of the 167 apples (12.0%) 
with browning were misclassified as good fruit. 
Different factors contributed to the errors of clas- 
sification. Some of the apples had large bruises in 
the vicinity of the region where the measurements 
were taken. Due to light scattering within the ap- 
ple. browning within bruised tissue had a similar 
effect on the spectrum when the bruise was located 
within 45° of the site. Five of the sixteen good apples 
that were misclassified had large bruises. Classifying 
defective apples as good could be very costly. For 
the 20 apples identified as good, only 2 apples had 
a moderate browning score. The remaining 18 ap- 
ples had very slight browning in a small area away 
from the site where the measurement was taken. 
Very slight browning is very difficult to detect if the 
damaged tissue is not located in the vicinity of the 
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measurement. A single measurement site was capa- 
ble of detecting small areas of brown ussue provided 
the defective region was within 45° of the measure- 
ment site. By increasing the cut-off between classes, 
apples with slight browning would be correctly clas- 
sified at the expense of increasing the percentage of 
good apples that would be rejected. 

Transmittance is a non-destructive technique for 
identifying apples with internal breakdown; how- 
ever, the technique is difficult to implement into an 
on-line inspection system. The integration time (5 s) 
required to improve the signal-to-noise ratio in the 
signal is too long for real-time inspection that re- 
quires less than 100 ms to measure the product. This 
time could be reduced by increasing the output of the 
light source; however, an increase in the output of the 
lamp causes an increase in the heat produced. There 
are techniques available to block the thermal energy 
while passing the shorter wavelengths. For apples, 
light must enter through the calyx end. Sufficient 
light transmittance by scattering was achieved only 
when the calyx end of the apple was directly illu- 
minated. Differences between classes were reduced 
when the cheek of the apple was illuminated and the 
energy exiting the fruit was measured at the calyx 
(unpublished data, 1994). The integration time had 
to be increased to 10 s to improve the signal-to-noise 
ratio. Without knowing the orientation of the apple, 
this technique would not be reliable for detecting 
internal breakdown in apples. 


4. Conclusions 


Apples with internal breakdown were distin- 
guished from good apples using a ratio of the trans- 
mittance at 720 and 810 nm. Setting a threshold of 
1.0 for the ratio, only 6.3% of the good apples were 
misclassified. Bruises on these apples were the ma- 
jor contributor toward misclassifying good apples. 
Over 12.0% of the defective apples were classified 


as good; however, the tissue in most of these apples 
only had a brown tint or the browning was confined 
to a small area away from the measurement site. 
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Abstract 


A fruit or other produce orienting device, particularly suited for orienting apples, orients a piece of fruit 

with its stem axis horizontal so that it can undergo quality inspections. The device employs a cylindrical 

drive roller which engages the fruit on one side, and causes it to rotate, and a pair of freely rotating 
orienting rollers which engage the fruit across from the drive roller, and cause the fruit to achieve the 
desired orientation as it rotates. The first orienting roller is cone shaped with a tapered flat or concave 
surface so that it will steer the fruit toward the second, opposing roller, which has a thin disk shaped 
portion with a front surface that engages the fruit. The front surface of the disk shaped portion tends to 
align itself with the flat edge of the fruit at its stem or calyx end, and this causes the rotating fruit to be 
oriented quickly with its stem axis parallel to the roller axes, and then maintains the fruit in the correct 
| orientation once it is achieved. 
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This application is a continuation-in-part of application No. 08/491,805, filed Jun. 19, 1995, now 
| abandoned. 





| What is claimed is: 


1. A device for orienting generally spherical shaped objects having a longitudinal axis comprising: 
| a) a drive roller for engaging and rotating an object to be oriented; 


! b) a first orienting roller spaced from said drive roller and positioned parallel thereto, said first orienting 
Toller having an outer end diameter larger than an inner end diameter, and an outer periphery thereof 
| forming a tapered object engaging surface; and 


_¢)asecond orienting roller spaced from said first orienting roller, said second orienting roller having a 
_ disk shaped portion with a front surface for engaging an object to be oriented; 


_ wherein, said drive roller and said first and second orienting rollers are positioned to engage a generally 
spherical object in three locations, and cause orientation of the object with its longitudinal axis in a 
generally horizontal direction. 





2. The object orienting device of claim 1, wherein said drive roller is disposed for rotation on a first shaft, 
_and said first and second orienting rollers are disposed for rotation on a second shaft which is spaced 
) from, and parallel to, said first shaft. 


3. The object orienting device of claim 2, wherein said first and second shafts are spaced from one 
another by approximately 4 to 5 inches. 


4. The object orienting device of claim 2, wherein said first orienting roller has an inner end diameter of 
_ approximately 2 inches. 


_5. The object orienting device of claim 4, wherein said first orienting roller has an outer end diameter of 
_ approximately 3 inches. 


6. The object orienting device of claim 2, wherein said first and second orienting rollers are spring loaded 
_0n said second shaft toward each other. 


| 7. The object orienting device of claim 2, wherein said first orienting roller has a concave outer surface. 


8 A method for orienting generally spherical shaped objects so that a longitudinal axis of each of said 
Objects is disposed in a generally horizontal position comprising the steps of: 


a) providing a rotatable drive roller for engaging and rotating an object to be oriented; 








6/27/01 2:49 PM 


QCA 9" AAA OT Ah BORLA OULERER., ante deeigeene 
























; > i 
uy) 2081 ST mat Lalt .*Dg, ena a silane areal a soa at 


va 
a ; 


Sm —— oe a o— an op?) (eas = — 
« 
aim) 


— ee ee SIE OS ee ee 


ynipniqince auc (uetuttondl s erved eneito bequrt egies ls 


nd 0) hwide ae yaitaton baiy 

ro mr: 
awe s 

v4 wy wrt bias vole ily latiexss lien ccna bis ‘tehon svnb bisa mot besege 1 
rj4/ rpacha ; Jeluo 6 Boe Solomect > tein ea aay w;utal vases ba 


«obi sasha pnigegra i do 


i” = : 


tueviters: Beovee bia “collet gniineie 210 bine mot? begage tener ae 


arsine 9d 0} jostle og greets tot Susthie inate dine " 
ee 


7 7 i 
Ot & 92eg\~ SUSIEZOg A = yoo git vstrd bnasee Ban tei ce Deis y vit 


P ol ti dtiw doeido- oc) Jo pottetnenoe 4aug0-bne 2nonAgol ge 


srt ‘ 


pene ul witeicd WN beaouab a: wilco ovr Bim dist . f siviy Yo 9290 


) sgictw Mele bagsse 4 to ousIOT KA ba qe ib sur xystion yetinghe by 
Flea veal Sn 


ji) Seanad ote eRtede bnogse bee nalt bine neotiw’ S misio ID soiveb net 
edn 2 oj & vin 


1, New ietns os ead tollot auitmens Jat Died shoei S romia te coi 
eu > ate fy ead isiloa golinai Jeu bise nisi  ntlahe 36 


babs inge ais evllon enittend bnosse ons 240 hue ieee a Hal. 
id 
is ue ereorne » ue fl lov patra raved. tas wna 

om - 
bike to doses le eos Linibongaol a tat on erssige Ga oe siviqga, 


te agate ott eee 7 sito 













tates Patent: 5,855,270 http://164.195.100.11/netacgi/nph-Parser?...5855270'. WKU.&OS=PN/5855270&RS=PN/5855270 
_ b) providing first and second spaced orienting rollers for engaging and orienting an object being rotated 
_ by said drive roller, said first and second orienting rollers being positioned for rotation about a common 
axis that is spaced from and parallel to a rotational axis of said drive roller, said first orienting roller 
_having a tapered outer surface with an inner end diameter smaller than an outer end diameter, and said 


second orienting roller including a disk shaped portion with a flat front surface facing said first orienting 
roller; 


c) engaging and rotating an object to be oriented with said drive roller; and 


_d) simultaneously engaging said object with said tapered outer surface of said first orienting roller and 
said flat front surface of said second orienting roller, wherein said tapered outer surface tends to direct 
said object into said flat front surface, and thereby cause orientation of said object with its longitudinal 


| Description 





| BACKGROUND OF THE INVENTION 


_ The present invention relates in general to a fruit orienting device for use with automatic sorting 
_ equipment that properly positions fruit or other produce, such as apples, for quality inspections. 


Considerable effort has been placed on developing nondestructive sensor techniques for sensing the 
internal and surface quality of apples. Examples of these techniques include optical imaging systems for 
detecting bruises and other surface defects on the apples, and firmness testing devices which determine 
the firmness of an apple by tapping its surface, and sensing resulting acoustic vibrations therein. To work 
| properly, both of these techniques require that the tests be performed on the apple's cheek. In the optical 
imaging technique, this requires that the imaging system be able to discriminate between the bruises or 
other surface defects, and the apple's stem or calyx. 





aoe 


One known technique for discriminating between surface defects and the stem or calyx is to employ 
complicated algorithms which can differentiate the concave surfaces around the stem/calyx ends from 
blemishes on the rest of the convex apple surface. Unfortunately, the use of such complicated algorithms 
_ slows down the apple sorting process, and this is unacceptable because the fruit packing industry requests 
that apples must be sorted at rates of 6-10 apples per second. 


_ Amore promising solution to this problem is to orient the apples prior to being imaged so that only the 

| apple cheeks will be viewed by the imaging optics. This usually necessitates that the apples be oriented on 
_ aconveying device with their stem axes horizontal so that the apples can be rolled beneath the imaging 
Optics, first for a preinspection for proper orientation with a reroute of improperly oriented fruit back to 
the beginning of the orientation process, and then for an inspection of the apple cheeks for bruises and 
other surface defects. This appears to be the most promising technique, for it eliminates the need for time 
| consuming complex algorithms. In addition, this technique must be employed with firmness measuring 

| devices and other quality measuring devices since such devices do not employ any optical means by which 


| the cheek of the apple can be discriminated from the stem or calyx. 


Unfortunately, all previously known techniques for orienting apples so that their stem axes are horizontal 
' do not reliably orient the apples in the desired manner. For example, one known device employs a 


' Plurality of rollers which contact an apple at three locations on its surface. One of the rollers is a 
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cylindrical roller that is driven to cause the apple to rotate so that its entire surface can be viewed by the 

_ imaging system. The other two rollers are positioned adjacent to the cylindrical roller on a common shaft, 
| and are cone shaped with tapered apple engaging surfaces that tend to hold the apple between these two 
rollers. Although this device is capable of orienting various types of apples with their stem axes in a 
horizontal direction, it does not do so quickly and dependably, and often cannot maintain the proper 
orientation if and when it is achieved. 


SUMMARY OF THE INVENTION 


In view of the foregoing, it is the object of the present invention to provide an improved fruit orienting 
device which can quickly and dependably orient pieces of fruit, particularly apples, with their stem axes in 
a horizontal direction. 


_ This and other objects of the invention are achieved through provision of an orienting device which 
comprises a drive roller for engaging a generally spherical piece of fruit or other produce on one side and 
| causing it to rotate, and a pair of freely rotating orienting rollers that are positioned adjacent the drive 

| roller in such a manner that they also engage the fruit. The first of the orienting rollers is cone shaped 
with one end diameter being larger than its other end diameter, thereby forming a tapered outer engaging 
| surface that tends to direct the fruit into engagement with the second roller. Preferably, the outer surface 
| of the roller is concave to better accommodate the spherical shape of the object (e.g. fruit or other 

_ produce) to be oriented, although the outer surface can also be flat and still produce good results. The 

_ second roller is disk shaped with a front engaging surface that contacts the flat edge of the fruit adjacent 
_ one of its ends. Both of the orienting rollers are preferably spring biased toward one another to allow 

| lateral movement for varying sizes of fruit. 








| In contrast with the previously discussed known apple orienting device in which the two orienting rollers 
| both comprise opposed cone shaped rollers, the use of the disk shaped roller in conjunction with the 

| roller having the tapered concave or flat engaging surface in the present invention results in substantially 
| improved orientation speed, accuracy and maintenance. This is because the disk roller's front surface 

| forms a contact surface which can align itself with the fruit edge at the stem/calyx ends, thus providing 

| guick horizontal orientation of the fruit, and stability to the revolving fruit once oriented. 


BRIEF DESCRIPTION OF THE DRAWINGS 
The features and advantages of the present invention will become more apparent from the following 
| detailed description of a preferred embodiment thereof, taken in conjunction with the accompanying 


| drawings in which: 


| FIG. 1A is a top view of a fruit or other produce orienting device constructed in accordance with a 
| preferred embodiment of the present invention; 


FIG. 1B is a front elevation of the fruit orienting device of FIG. 1A showing its rollers engaging a 
| properly oriented apple; 


FIG. 2 is a side view of the device of FIGS. 1A and 1B; 


FIG. 3 is another top view of the device of FIG. 1A showing its rollers engaging a properly oriented 
| apple; 


FIG. 4 is a schematic illustration of a fruit conveyer employing a plurality of the orienting devices of 
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FIGS. 1-3; 


FIGS. 5A and 5B are graphs illustrating experimental results obtained using the concave surfaced roller 

variation of the preferred embodiment of the present invention to orient apples, with FIG. 5A illustrating 
the percentage of apples properly oriented as a function of time for five different types of size 128 apples 
being rotated at 60 rpm, and FIG. 5B illustrating the percentage of apples properly oriented as a function 
of time for five different types of size 88 apples rotated at a speed of 60 rpm; 


FIGS. 6A and 6B are graphs illustrating additional experimental results obtained using the concave 
surfaced roller variation of the preferred embodiment of the present invention to orient apples, with FIG. 
6A illustrating the percentage of apples oriented from an axis of rotation perpendicular to the stem/calyx 
axis to an axis of rotation about the stem/calyx axis as a function of time for five different types of size 
128 apples, and FIG. 6B illustrating the same results for five different types of size 88 apples; 


FIGS. 7 and 8 are graphs illustrating experimental results obtained using flat surfaced roller variation of 
the preferred embodiment of the present invention to orient apples, with FIG. 7 illustrating the percentage 
of apples properly oriented as a function of time for five different types of apples and FIG. 8 illustrating 
the percentage of apples maintaining their proper orientation as a function of time; and 


_ FIGS. 9 and 10 are graphs illustrating experimental results obtained using a known apple orienting 

_ device, with FIG. 9 illustrating the percent of apples correctly oriented as a function of time for five 
different types of apples, and FIG. 10 illustrating the percent of apples maintained in the correct 
orientation as a function of time. 


DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 





Turning now to a detailed consideration of a preferred embodiment of the present invention, FIGS. 1-3 
illustrate a fruit or other produce orienting device 10 comprising a cylindrical drive roller 12 which is 

_ Totated by a power source, such as a motor (not shown), through a drive belt 14. The cylindrical drive 
Toller 12 preferably has a soft resilient (e.g. padded) surface, and is mounted on a first horizontal shaft 16. 
_ Preferably it has a length (e.g., 3.75 inches) that is somewhat greater than the typical height of the fruit to 
_ be oriented (e.g., an apple). 


First and second orienting rollers 18 and 20 are mounted for free rotation on a second horizontal shaft 22, 

which is positioned parallel to the first shaft 16, and is spaced therefrom by a distance of preferably 

_ approximately 4-5 inches. The first orienting roller 18 is cone shaped with an inner end diameter 24 that 
is preferably approximately 2 inches, and an outer end diameter 26 that is preferably larger, for example, 

_ approximately 3 inches, thereby forming a tapered peripheral or outer engaging surface 27 which is 
tapered in the direction of the second orienting roller 20. The thickness of the cone shaped roller 18 is 
preferably approximately 1 inch. Preferably, the outer engaging surface 27 is concave in shape as 
illustrated to better accommodate the spherical shape of the fruit or other produce to be oriented, thereby 
further improving the orienting performance of the device 10 when used to orient cultivars, for example, 
which are difficult to orient. Alternatively, the outer engaging surface 27 can be flat as illustrated by the 
dashed lines in FIG. 1A, and still provide good results as discussed below in conjunction with FIGS. 7 
and 8. 


The second orienting roller 20 has a disk shaped portion 28 having a diameter approximately equal to that 
of the outer diameter 26 of the cone shaped roller 18. The disk portion 28 includes a rounded edge 30, 
and has a thickness of approximately 3/16 inches. A front, flat engaging surface 32 is formed generally 
perpendicular to the axis of the second horizontal shaft 22 for engaging the surface of the fruit to be 
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| oriented. Preferably, both of the rollers 18 and 20 are spring loaded toward each other by a pair of 
compression springs 33 of equal length and spring rate which allow lateral movement for varying sizes of 
| fruit. 


As illustrated in FIGS. 2 and 3, the particular spacing and arrangement of the drive roller 12 and orienting 
| tollers 18 and 20, enables the drive roller 12 to engage an apple 34 or other generally similar ‘shaped 

_ object to be oriented and rotate the same, while the orienting rollers 18 and 20 engage the apple 34 ona 

| generally opposite side thereof. As also illustrated in FIG. 3, the front engaging surface 32 of the disk 

_ shaped portion 28 is designed to engage one of the flat shoulder portions 35 of the apple 34 positioned 
adjacent its stem or calyx ends. 


The two roller shafts 16 and 22 are fixed at their opposite ends to first and second conveyor chains 38. 

As illustrated in FIG. 4, a plurality of the devices 10 are preferably incorporated into a conveying 

_ apparatus 40 which conveys a plurality of pieces of fruit to be inspected past a sensor device 44, which by 
way of example can be an imaging device for visually inspecting the apple cheeks, a firmness testing 
device or any other device for sensing the apple's condition. 


| 
| 
| 
) 
| 


_In operation, randomly oriented apples are conveyed onto a plurality of the devices 10, where they are 
_ each engaged at three locations on their surfaces by the drive roller 12 and the orienting rollers 18 and 20 
_ of their corresponding device. As the apple is rotated by the drive roller 12, the cone shaped roller 18 
tends to move the apple into engagement with the disk shaped roller 20. When this occurs, the apple has 
a tendency to seek a position where its largest diameter (its width) engages the drive roller 12 and one of 
tts ends (stem or calyx) contacts the disk shaped roller 20, with the axis of rotation for the fruit lying 
along its shortest dimension, which is the apple's height. Because of these tendencies, the front engaging 
surface 32 of the disk shaped roller 20 tends to align itself with the flat edge of the apple at either the 
stem or calyx ends, and provides stability to the revolving apple once it is oriented. 


_ Experiments were conducted to compare the orientation success rate of the preferred embodiment of the 
_ present invention using both concave and flat surfaced orienting rollers with that of the previously 

_ discussed known apple orienting device. The results of these experiments are illustrated in FIGS. 5-10, 
with FIGS. 5A, 5B, 6A and 6B illustrating the results obtained with the preferred embodiment of the 

| present invention using a concave orienting roller, FIGS. 7 and 8 illustrating the results obtained with the 
| preferred embodiment of the present invention using a flat surfaced orienting roller, and FIGS. 9 and 10 
illustrating the results obtained using the known orienting device. 


FIGS. 5A and 5B illustrate that the orienting device 10 with the concave surfaced orienting roller was 
able to orient 95 to 100% of various sizes of Cortland, Empire, Fuji, Jonagold, McIntosh and Rome 
apples within approximately 0.2 minutes. FIGS. 6A and 6B illustrate the experimental results obtained 
_ with various sizes of apples that are known to be difficult to orient because of a greater variation in both 
_ size and shape, including Golden Delicious, Granny Smith, Crispin, Red Delicious and small Galla apples. 
With the exception of large Granny Smith and Crispin apples, successful orientation was achieved 60 to 
_ 82 percent of the time. 


| Experiments were also conducted using the flat surfaced orienting roller variation of the preferred 
embodiment which performs well when used to orient the less difficult to orient types of apples. As 
illustrated in FIG. 7, the flat surfaced roller variation correctly orients the stem/calyx axis into the 
horizontal position (parallel to the axis of roller rotation) of Cortland, Jonagold, Idared and Empire 

| apples 88 to 100% of the time, and within 12 seconds. Once oriented, the device 10 is virtually 100% 
successful at maintaining the proper orientation of the apples as illustrated in FIG. 8. 





6/27/01 2:49 PM 


eee ae a ae 


si wilit b ri = d “2 : avin ort + Kw Gy 51a: ‘e 44 ipA a9iveb ‘ler 
sic i ut: padW O° wllor booed any ot) dew Peemisanges ont lag 


inne Biiuaes Letreintiksa xo Srl} eistaulli da bes Ad 2D wenn =o 


£00 hovsiriss caw medsIinene lifeessowe asiqge niqeltD ban tines 4 









































a ‘t7? Croets fret nena ti OOF Poi be rwqnd 


» AMEN file tone Geto esigqern! BAR OF 0% Dag BI aration, ott td io dyed Y 
a 
2 gpa 1 wercevnen Inn wells dokiw sun gahge bag cigneepe te fee 
y 
a 
mw . 
ur Sf yell vied ovlt Yo tier: bes ertiose elucieg oat ee 
Vill Sp “Wre: if mT. SVIGS (ie < Leas ¢ j Si mallet aT afi Sxde 
nie Bl ensBee vratnavic ots oleh yertine net On} ror bine be 
ah wer elt .€ 214 viet betereuil nels 2A Aussi sbi s 
meerrod twhluwers Ist) ods To ao syeueS OF a & 
ebao xyiho 16 
a 


: 
ban iziit of thes stieayne tort Jc baa ot Las M1 

103 E &h fie! nj viderstown 97a Of esoevek ott To v ng af Om Mo 
guriee & jan beloscamn od of fiuttto cosig to yield 4 vn 

; , 7 Siyis on! ges yi adety Ww @orvab ONES te od m1 ve 
cite 4 SIGQE ot Onis IOI yoveB: 


S 


ie s ne bo yeencs SUR See LOIN! ay One 


itor evink adi vd ssuetue ell ao encilsapl se uit ’ 


st t (apr ) 299rtre1) own Bn saw I nang ol 
- ‘ } pruht i g os fii ; } rt i bs Lette m he - fy v roe cre {x 2% 
raf zee! % avuaoet! forest 2 ea 17 } Sieyy Linon I 


mf . 

i ii. tie iva 

: ' fis Au J ait : > Geepa a 
i = 5 ’ Ss rv ~ 7 

IO Al HID DIGG fF virege4? Sci OT Virutisie E DPT , 


OST 1.Of) 10 DIRT BRSSOUE MOM EINSTIO Sile-ore a i beiopaed or 
i : 
i. ii loi ufaias efnue teh bes oviewie®d ane Gril ae 


£ prsdes s2atit to ales oT sorveb anna oft 16 * 
ibodins herine afidive baniedg efivesy od) gadsdeult GO tee os 
wiiairlo ative od! qniseweul: 2 bee © SOP weilot aotigano svgonos ane 
2 2017 bas sll sates bsadtrue tall & ghia notnerth ey {20 3 
soiveh andnene mwona af shay ae he 
oa silk 


~ Lh. OL gt 


t fuer boosie Svnonon Gu Aiter 91 vaiveb grilinorie Si iets 
sta bas fecinisM Dleganol 7A Wig bnsineD 25K eure 


AAUPCeY ~otewte 4 to seamed tasiio- ol jodi od ot cesomnl sta tbe 


14 allot) lene Woe cuddles bell migeitD die winere oink ooo 


a prniae, 





Wie 


boridierg sd? Ws mcitahes ‘icine haba we gives be betoubs 
2A eslate lo esgyl inate & priiredritekars moe ¥ ey 


een ee 


silt eal: pect mebar pee a coy: eilior boca 
sien bak bewht binganol lov ta civ sh 

@00LY ‘Lamtiy i 0) splot oth patric 6 son els ‘t nithive b 
8 ONT ntboem slag at 


; 1 


tates Patent: 5,855,270 http://164.195.100.11/netacgi/nph-Parser?...5855270'. WKU.&OS=PN/5855270&RS=PN/5855270 


In contrast, experiments with the known orienting device employing a pair of opposed cone shaped 
rollers, indicate in FIG. 9 that this technique is successful at properly orienting the various types of apples 
73-95% of the time after a period of 16-18 seconds. In addition, once the apples were oriented, many of 
them lost their orientation within 10 seconds as illustrated in FIG. 10. 


It should be pointed out that neither the flat surfaced roller embodiment of the present invention nor the 

_known orienting device were particularly successful at orienting Red Delicious apples as can be noted 

from a careful inspection of the graphs in FIGS. 7-10. This is because many Red Delicious apples tend to 

have an elongated pear shape with a much different fruit diameter at the stem end than at the calyx end, 

and the calyx end of this type of Red Delicious apple does not present a good shoulder surface for the flat 
surface 32 of the disk roller 20 to engage. However, as discussed above in conjunction with FIGS. 6A 

_and 6B, the concave surfaced roller variation of the preferred embodiment works considerably better at 
orienting Red Delicious, and other difficult to orient, apples. 


_ In conclusion, the present invention provides an improved fruit orienting device that is particularly suited 
| for orienting fruit or other produce, such as apples, about to undergo a quality inspection. The device 

| provides a substantially greater orientation success rate, achieves the proper orientation in less time and 

_ better maintains the orientation once achieved than any previously known fruit orientation devices. This is 
| particularly significant when the device is employed with automatic apple sorting devices which must 

| inspect large quantities of apples for bruises and other defects, and sort the apples accordingly in a fast, 

| efficient and dependable manner. 


Although the present invention has been disclosed in terms of a preferred embodiment, it will be 

| understood that numerous other modifications and variations could be made thereto without departing 

| from the scope of the invention as defined in the following claims. For example, although the invention is 
designed specifically for orienting fruit, specifically apples, it will be understood that it can also be 

| employed for orienting other similarly shaped produce, such as vegetables. In fact, the present invention 

| can be employed for orienting any generally spherical shaped object having some type of asymmetry, such 
asa stem, defining a longitudinal axis. 
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Summary: A design and implementation of a multispectral inspection station is presented. Apple images at 
four different wavelengths are captured through a common aperture. Design features include 
lighting, chromatic aberration correction, flat-field correction, apple presentation and conveyance, 
and defect segmentation. Example images at 4 wavelengths are shown with the results of defect 
segmentation. 
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ABSTRACT 


A design and its implementation are presented for an automatic apple defect sorting 
system utilizing elements from 25 years of past research. The design included methods 
for apple orientation, conveyance, and presentation to a camera at five apples per 
second. 


Special optics and filters allow image capture of apple reflectance using three 
wavelength bands with center wavelengths of 540nm, 750nm, and 960nm and 
bandwidths of approximately 60nm. A fourth image captured included all visible and 
near infrared wavelengths that the camera array could detect. The camera captured all 
four images simultaneously through a common aperture device with selected optical 
filters added to obtain the desired wavelengths. 


Lighting was designed to provide diffuse lighting at the required wavelengths to match 
the relative response of optics, filters, camera, and apple reflectance. This required the 
addition of compact fluorescent lamps to increase light for the 540nm wavelength. 


A method for defining and capturing 6 regions of interest per apple as multiple apples 
conveyed laterally and rotated through the camera’s field of view was presented. 


Imaging software captured the regions of interest and applied flat-field correction to 
correct apple reflectance variation caused by surface curvature and lighting variation 
across the field of view. Average reflectance intensity was used to set threshold levels 
for defect segmentation. The size and shape of the segmented pixel clusters were used 
to select clusters that represented defects. 


Example images of the four wavelength bands and the segmented defects were © 
presented. 


BACKGROUND 


Past research has developed many of the elements required to construct a high-speed 
system for apple defect detection. Elements include apple orientation, stem/calyx 
identification, lighting, defining reflectance spectral characteristics, camera/optics 
design, and image processing. 


Apple Orientation: 


Placing the stem/calyx ends of an apple in a known position eliminates the need to 
locate them during the defect detection process. The stem/calyx regions can appear 
similar to defects on the apple surface making it difficult to differentiate from defects. 
One device consisted of a rotating inverted conical shaped transport cup with a hole in 
the bottom through which a small wheel rotated while contacting the fruit. When the 
stem or calyx moved to a position over the wheel, contact with the apple surface ended 
leaving the apple oriented with the stem/calyx in a vertical position ( Keesling, 1965, 


Rehkugler et. al. 1976). The apple was in the bottom of the cup making viewing of the 
apple surface difficult. 


Current bicone roller conveyors were found to orient Red Delicious apples from 
Washington State to an axis of rotation passing through both the stem and calyx. Most 
varieties grown in other parts of the country tended to rotate about an axis 
perpendicular to the stem/calyx axis. A new device consisting of a vertical disk and a 
cone roller with a concave shape spring loaded towards each other in combination with 
a powered rotating padded cylindrical roller was built to convey apples while they 
oriented ( Throop et. al., 1995, 1997, 1999). The device oriented 95% to 100% of 
Cortland, Empire, Jonagold, Mcintosh, Rome, and small Fuji. Golden Delicious, Granny 
Smith, Crispin, and Gala oriented 60% to 82% of the time. The resulting apple position 
of the apple rotating about an axis passing through the stem/calyx regions was 
conducive for viewing the apple surface for defects. 


Stem/Calyx Identification: 
A method using two images: one image of the apple using diffuse lighting and a second 


using structured light distinguished the stem and/or calyx from other defects based on 
the curvature of the laser lines. However, no information was provided to describe 
conditions causing the algorithm to fail in the identification process (Yang, 1993,1996). 
A study using a structured light consisting of nine parallel lines of light found that 
identification errors increased to as high as 20% as the stem/calyx position moved 
towards the outer edges of the apple image (Campins et. al., 1997). 


Lighting: 

Lighting for manual sorting of apples has been and continues to be an issue in packing 
houses. Having enough light that is both diffuse to reduce glare and with the correct 
wavelengths aids workers in the sorting process (Brown, 1991). Automatic optical 
inspection of fruits and vegetables requires lighting system design that match the 
reflectance properties of the physiological character of the desired product and the 
performance characteristics of the optical system elements (Affeldt et. al., 1993). 


Reflectance Spectral Characteristics: 
Knowledge of the spectral reflectance properties of apples and their many defects 


define the design parameters of the lighting and the camera/optic systems. The 
reflectance spectra of 21 defects found on 10 apple cultivars were captured for 
wavelengths between 460nm and 1030nm in 10nm increments using an electrically 
adjustable filter. The Mahalanobis distance was used as a measure to find the greatest 
contrast between defect and undamaged tissue reflectance. Three wavelength bands 
60nm wide with center wavelengths of 540nm, 740nm, and 1010nm provided that 
contrast for most defects (Aneshansley et. al. 1997, Throop et. al., 1997). The visible 
images captured for these studies did not include a near-infrared blocking filter for the 
adjustable visible filter. A new study undertaken in 1998-1999 has found a fourth 
wavelength band at 650nm that should be included (Unpublished Data, 1999). Images 
in this paper will not include this wavelength since apples with appropriate defects are 
not available until the 1999 harvest. 








Camera/Optics Design: 


The use of the response characteristics of a silicon sensor array in combination with 
filters to form an optical system sensitive to particular wavelengths has been used 
experimentally for bruise detection (Graf, 1981,1982). Optimizing lighting with respect to 
the camera response can maximize contrast between bruised and undamaged regions 
(Throop et. al., 1993). Knowledge of light wavelength emission and reflectance spectral 
characteristics is required to select the appropriate camera array response and filter 
combination. Transmission characteristics of the filter and lens also are important in 
optimizing sensory response, that is, matching the reflectance properties of the 
physiological character of the product and the performance characteristics of the 
optical/camera system (Affeldt et. al., 1993). 


Image Processing: 
Considerable effort has been used to expedite the segmentation of defects from the 


undamaged background of apple images. Most effort has been directed at bruise 
detection where the change in gray level intensity between bruised and undamaged 
tissue is less than 10%. A multivariable statistical classifier of each pixel in an apple 
image classified bruised from undamaged tissue (Graf et. al., 1981, 1982). A decrease 
in processing time and improved performance was found for a non-statistical processing 
approach. A low-pass filter was applied to the original image. The original gray level 
image was subtracted from the low-pass filtered image and the resulting image is 
thresholded to form a binary image. A form of contour - following is applied to each 
segmented cluster of pixels and a dimensionless measure of shape called the thinness 
ratio is found. Thresholding the thinness ratio selects the pixel clusters from the 
segmented image that are counted as damage (Throop, et. al., 1988). This processing 
method was applied to other defects besides bruising (Rehkugler et. al., 1989). 
Investigation of three methods of contour following for computational efficiency and 
accuracy of predicted pixel cluster area and perimeter length found that the method 
presently selected was best (Sagi et. al., 1991). Further research added processes to 
segment stored bruises whose reflectance had changed to greater than undamaged 
tissue. An experimental method for estimating gray level threshold values and minimum 
values for shape factor was found (Throop et. al., 1995). Images of the same apple at 
three wavelength bands 50nm — 60nm in width with center wavelength of 540nm, 
740nm, and 1010nm were processed in a similar manner to segment 22 different 
defects found on 10 different apple cultivars (Throop et. al., 1997). 


Combining the design elements into a machine capable of sorting apple defects at 5 
apples per second is the subject of this paper. 


OBJECTIVES 


1. Present a design and its implementation for the inspection of apples incorporating the 
elements from past research with present day technology. 


2. Discuss problems encountered in this process. 





Figure 1. System 1 (right) using conventional bicone roller conveyor and system 2 (left) 
using an experimental conveyor. 


MATERIALS AND METHODS 


Apple Conveyance and Presentation: 


Two conveying and orienting systems were developed to present apples to the camera 
in an oriented position with an axis of rotation common with an axis passing through the 
stem/calyx ends. Two systems were designed to accommodate orientation for different 
apple cultivar shapes. 


System 1 used a typical bicone roller conveyor that orients elongated apples such as 
Red Delicious from Washington State. Agri-Tech Inc., Woodstock, VA, loaned the 
device to the research project (Figure 1). Modifications to the system included variable 
speed control of the linear travel speed of the conveyor and the rotational speed of the 
bicone rollers(253G-200E, Dart Control Inc., 62410, Datyon Electric Co.). Fiber optic 
sensors were added to the conveyor to trigger the camera when the apple was correctly 
positioned( FE5F-1MC6S-M, Honeywell Microswitch, Minneapolis, MN). 


System 2 was built at Cornell University and could orient York, Stayman, Cortland, 
Empire, Fuji, Jonagold, McIntosh and Rome apples. It incorporated a new device 
consisting of a vertical disk and a cone roller with a concave shape spring loaded 
towards each other in combination with a powered rotating padded cylindrical roller 
described in the background (Figure 1). Variable speed control of the linear travel speed 
of the conveyor, variable speed control of the rotational speed of the bicone rollers, and 
fiber optic sensors similar to the first system were also added. 


Image Capture System: 


A scientific grade digital camera was selected that images at 30 frames per second with 
Spatial resolutions of 1024x1024x12 bits (SMD-1M30, Silicon Mountain Design, 
Colorado Springs, CO). The camera was interfaced to the PCI computer bus by an 
image capture card with 4 MB VRAM (Raptor, Bitflow Inc., Woburn, MA). Data enters a 
32-bit RS422 digital port that can be externally triggered at up to 40MHz. Dual ported 4 
MB memory allows the host to access data during acquisition. The host was a 533MHz 
clocked 21164A Alpha processor equipped desktop computer using a Durango || 
motherboard with 4 MB cache and 512 MB memory with a Windows NT operating 
system (Aspen Systems Inc., Wheat Ridge, CO). Image capture and processing 
software was a custom version of QUANTIM software (ZEDEC Technologies, 
Morrisville, NC). 


= 





Figure 2. Common aperture device with a 6.5mm lens mounted on the front and the 
camera behind. 


Filters and Optics: 


A common aperture device was placed before the camera to project four spectral 
images of the apple on the single focal plane of the camera array (MSAI-04V, Optical 
Insights, LLC, Tucson, AZ). This device enabled simultaneous capture of spectrally 
discrete real — time images with a single camera (Figure 2). A 6.5mm high resolution 
Monofocal CCTV C-mount lens was mounted in front of the common aperture device 
(23FM65, Tamaron Industries, Farmingdale, NY). An F-mount to C-mount adapter 
coupled the device to the camera. The filter plate was fitted with four 2.54cm diameter 
holes to fit interference filters for the selected wavelength bands. Filter 1 was a green 
additive dichroic with 50% transmission starting at 505nm and ending at 575nm (CA- 
550-F, Corian Corp., Franklin, MA). Since this filter passes near-infrared, an infrared 


cutoff filter was added (D53710, Edmund Scientic, Barrington, NJ). Filter 2 had a center 
wavelength of 750nm and a bandwidth of 65nm (750 IF 65, Cheshire County Optical 
Inc., Jaffrey, NH). Filter 3 was a long wave pass filter with a cut-on/off wavelength of 
950nm (950FH90-25, Andover Corp., Salem, NH). Filter 4 initially passed all 
wavelengths. Neutral density filters were added as needed to normalize image intensity 
for the different wavelengths (NDO10R/25, NDO50R/25, NDO70/25, Maier Photonics 
Inc., Manchester Center, VT)(03 FNG 040, Melles Griot, Irvine, CA). While design was 
underway for the machines, it was discovered that the visible wavelength data from the 
variable filter for measuring the spectral reflectance characteristics included near- 
infrared wavelengths. The previous test should have included a filter to block near 
infrared wavelengths. This information was not specified clearly in the filter 
documentation requiring a second set of images to be captured. A fourth wavelength 
band with a center wavelength of 650nm was discovered (Unpublished Data, 1999). In 
the future Filter 4 will have a center wavelength of 650nm and a 40nm bandwidth (650 
IF 40, Cheshire County Optical Inc., Jaffrey, NH). Filter 2 and filter 4 were ordered with 
painted edges in place of standard metal ring mountings. 


Lighting 


A custom designed fixture composed of five 100W soft white incandescent lamps 
(Osram Sylvania, Danvers, MA) alternately spaced between four compact fluorescent 
lamps (DULUX EL23W, Osram Sylvania, Danvers, MA). A diffuser cut from 1.59mm 
thick white teflon sheet (Teflon TFE, Ain Plastics, Mount Vernon, NY) was applied on 
the conveyor side of a pair of fixtures for each machine (Figure 3). The incandescent 
lamps were powered by variable rectified AC voltage. Constant 115 volts AC powered 
the compact fluorescent lamps. 





Figure 3. Custom lighting fixtures positioned on each side of the experimental conveyor. 





RESULTS AND DISCUSSION 


Relative System Response 


Lighting, filter selection, and camera response all had to be considered together in order 
to obtain high quality images at the desired wavelengths. The relative response of the 
camera, transmitance of the filters, and light output from the incandescent and compact 
fluorescent lamps were graphically plotted for wavelengths between 400nm to 1000nm 
(Figure 4). The fluorescent lamps were added to increase the amount of available light 
in the 540nm bandwidth to counter the low camera response and incandescent lamp 
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Figure 4. Graph showing the relative response for the different components that formed 
the imaging system. 


emission in that range of wavelengths. The incandescent lamp’s increasing emission as 
the wavelength increases supplied ample light for the 960nm wavelength even with the 
low camera response. 


Region of Interest 


Apples were conveyed through the camera field of view rotating about an axis passing 
through the stem/calyx ends. With the apples oriented to this position, it is possible to 
view most of the apple cheek with a single camera. This eliminates the stem/calyx ends 
from being identified as defects and enables placement of other cameras attuned to 
inspect the stem/calyx regions. Present research is currently focused on the apple 
cheek. 


As the rotating apple passes through the 355.6mm field of view, images of 6 regions of 
interest across the field of view and around the rotating apple’s perimeter are captured. 
The position of the six regions of interest must be established with respect to the 
conveyor position, the camera array position and the rotational speed of the apple. This 


positioning process starts with the spatial location of an optical sensor monitoring the 
apple’s position on the conveyor that provides an external trigger signal to the camera 
for image capture. Three white Delrin spheres 76.2mm diameter are placed adjacently 


750nm No Filter 





950nm 540nm 
Figure 5. Simultaneous capture by the camera array of the three Delrin spheres passing 
through the field of view for the four wavelength bandwidths. Starting at the upper left 
quadrant of the image and rotating clockwise, the center wavelengths are 750nm, all 
wavelengths, 540nm, and 950nm respectively. The six regions of interest are drawn 
automatically centered on the spheres. 


on the moving conveyor. Images of the spheres are captured as they passed through 
the camera's field of view at a distance of 578mm from the optics/camera (Figure 5). 
Software centers the six regions of interest on the Delrin spheres and the size of the 
regions are set based on the size of the apples to be inspected. The location of each 
region of interest is stored in a file and can be manually edited for fine-tuning if 
necessary. As apples pass through the camera field of view, pixel data from the six 
regions of interest are captured and displayed adjacent to each other. The resulting 
image depicts the reflectance for each wavelength band of the total apple surface 
excluding the stem/calyx ends. Although not implemented, the stem/calyx ends can be 
imaged by a single frame capture by separate miniature cameras during the time 
traveling past the initial camera. 


Images are captured of a Teflon sphere with a black tape stripe every 60 degrees. If the 
proper rotational speed is set, the black stripe will appear at approximately the same 
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location in each region of interest. With proper spatial registration for each image at 
different wavelengths, only black lines without white edges or white lines without black 


edges will appear in the images formed by subtraction of various image pairs (Figure 6). 
750nm 950nm-750nm No Filter No Filter-950nm 
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Figure 6. Image of Teflon sphere with tape stripe every 60 degrees checking for spatial 


and rotational alignment. 
Flat Field Correction 


An image of a white Teflon sphere 76.2mm diameter is captured for the four wavelength 
bandwidths. The image is used to manually place neutral density filters in combination 
with the bandpass filters making the gray level output for the Teflon reflectance 
normalized to 200. The gray level readings for a row of pixels across a region of interest 
for any wavelength band were found to be nearly uniform (Figure7). The gray level 
readings for a column of pixels read across all six regions of interest showed the 
curvature of the Teflon sphere as each region of interest boundary was crossed (Figure 
8). If it is assumed that the teflon sphere approximates the reflectance of undamaged 
tissue on apples, an inversion of the Teflon image can be added to every apple image 
captured for flat field correction. The gray level readings across 6 regions of interest for 
an apple image for the 750nm wavelength after applying flat field correction as 
described no longer show the curvature of the apple at the boundaries of the regions of 


interest (Figure 9). This simplifies the thresholding process while segmenting apple 
defects. 
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Figure 7. Plot of the camera output in gray level for a row of pixels across one region of 
interest. 
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Figure 8. Plot of the gray level intensity of a column of pixels selected across all 6 
regions of interest from an image captured of a teflon sphere. 
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Figure 9. Plot of the gray level intensity of a column of pixels selected across 6 regions 
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of interest captured of an apple with flat field correction applied by adding the pixel 
values of the same regions of an inverted image captured of a teflon sphere. 


special Optics Adjustments 


The special optics from Optical Insights were difficult to align and keep in alignment. 
The position of the images would not stay spatially matched between array quadrants 
and were difficult to readjust. The F-mount to C-mount adapter did not allow enough 
flexibility for alignment with the camera and was removed and replaced by two 
telescopes sleeves. The construction of the mounting box was poor and did not keep 
the vertical mounting plates holding the lenses and prism rigid and parallel. In addition, 
it was difficult to correctly position the filter plate when removed to add additional neutral 
density filters. The mounting box was reconstructed from aluminum plate with parallel 
slots machined to align the edges of the mounting plates. Adjustment screws in the x 
and y direction were added to center and lock the prism mounting plate and the filter 
holding plate in the optical path between the input and output lenses. 


Chromatic Aberration 


The index of refraction is a function of the materials forming the optical path and 
wavelength. As wavelength increases, the index of refraction decreases for the same 


750nm No Filter 750nm No Filter 
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Figure 10. Image of Teflon sphere with a black stripe before (left) and after (right) 
chromatic aberration correction. 


optical path. The index of refraction for the 540nm wavelength is higher, thus this image 
is focused closer to the rear lens of the special optics and further away from the camera 
array than images for 950nm wavelength (Figure 10 left). By adding elements that 
decrease the index of refraction for the shorter wavelengths, the focus point for the 
shorter wavelengths can be made to coincide with the 960nm wavelength (Figure 10 
right). 


Camera Adjustments 


The SMD-1M30 camera provided 12-bit intensity resolution while 8-bit resolution was 
considered adequate. If the exposure exceeded 255 gray levels (8-bit) the lower 8 bits 
of interest would be set to 0 and bit 9 would rise to 1. The result was that if the camera 
saturated higher than 8 bits the image would darken instead of remaining at a saturated 
gray level of 255. Silicon Mountain Design provided additional interface circuitry to hold 
the gray level at 255 for the saturated condition. They also factory adjusted the camera 
gain to maximize sensitivity and to match each camera used in system1 and system 2. 


Image Capture and Defect Seqmentation 


Images are captured and processed in a time period of 150ms. The machine with the 
commercial bicone roller conveyor can have as many as three apples in the camera’s 
355.6mm field of view during a single frame capture. This means that regions of interest 
for 3 different apples most be stored for every frame capture. Each apple passing 
through the field of view has a total of six regions of interest stored of its surface from 
six different frame captures. The machine with the experimental conveyor has a blank 
conveyor area between each apple reducing the flow rate by nearly one-half. 


The processing steps are as follows: 

. Capture Image. 

. Add or Subtract Combinations of Images for Different Wavelengths. 

. Find Average Gray Level. 

. Adding +/- Value to Average Gray Level to Set High/Low Intensity Threshold. 
. Threshold Defect. 

. Erode or Dilate as Required. 

. Measure Segmented Defects. 

. Threshold Defects Based on Size or Shape Factors. 

. Record to File Defect Size in Pixels or Percent of Total Surface Area. 


OONOaARWD = 


The processing steps are applied on images from any of the four wavelength bands. 
The processing can be tuned to look for expected specific defects or defects common 
on a particular cultivar. Some defects were easily segmented and were visible in more 


blush on its cheek, some defects in the 540nm wavelength band can be masked by the 
coloring. 


No Filter 950nm- 540nm 950nm- 
540nm 750nm 
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950nm = 950nm+ 750nm Segmented 
750nm Damage 


Figure 11. Images of the surface of a Golden Delicious apple with brown rot for the four 
wavelength bands, mathematical wavelength combinations, segmented damage, and 
visible wavelength views of each side of the apple (left). 


Bitter pit on Red Delicious is best processed from images from the 750nm wavelength 
band. It is practically invisible in both the 950nm and the 540nm wavelength bands 
(Figure 12). Studies undertaken in 1999 indicate that even better contrast of bitter pit 
damage from undamaged tissue can be expected for the 650nm wavelength. 


Bruising has the best contrast in the near-infrared wavelengths. The example shown is 
a Stayman apple removed from a bin right before processing. The bruise can be clearly 
seen in both the 750nm and 950nm images (Figure 13). For examples of other 
segmented defects using multiple wavelength images see Throop et. al., 1997. 


No Filter 950nm- 540nm 950nm- 
540nm 750nm 





950nm 950nm+ 750nm Segmented 
750nm Damage 


Figure 12. Images of the surface of a Red Delicious apple with bitter pit for the four 
wavelength bands, mathematical wavelength combinations, segmented damage, and 
visible wavelength views of each side of the apple (left). 


Future Challenges 


The addition of a user-friendly interface allowing the user to adjust sensitivity for the 
different threshold values while processing to detect the many different kinds of defects 
is necessary since some markets have stricter sorting requirements than others. Means 
for adjusting the region of interest size for different apple sizes and cultivars is 
necessary. More apple cultivars need to be oriented by redesign of the conveyor or 
stem/calyx identification needs to be added. 
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No Filter 950nm- 540nm 950nm- 
540nm 750nm 





950nm 950nm+ 750nm Segmented 
750nm Damage 


Figure 13. Images of the surface of a Stayman apple with bruising for the four 
wavelength bands, mathematical wavelength combinations, segmented damage, and 
visible wavelength views of each side of the apple (left). 


CONCLUSIONS 


The design and implementation of an inspection station to detect surface defects on 
apples is presented. 


The systems capture multiple images of each rotating apple as they are conveyed 
through the inspection station at up to 5 apples a second. Special optics produce four 


identical spatial subimages in each quadrant of the captured frame. Each subimage has 
different wavelength characteristics 


Concerns encountered in this process are discussed as follows: 


1. Matching relative response of camera, filters, optics, and lighting to a wide variety of 
apple cultivars and their defects. 
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2. Setting the regions of interest to capture the surface of an apple traveling laterally 
and rotating through the camera's field of view. 


3. Providing flat field correction to adjust for variation in reflectance caused by apple 
surface curvature. 


4. Chromatic aberration correction enabling focusing of multiple images captured 
through a common aperture at different wavelengths. 


5. Defect segmentation using features of reflectance for different wavelengths, 
thresholding based on average reflectance intensity, and segmented feature size and 
shape. 


Example apple images captured at four wavelength bands and a segmented image are 
presented. 
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